Day 2026-10-23 close-out: rulings D31-D36 in CONTEXT; STATUS (releases done, do list, D35 dated, D37/D38); register 191 -> 184 (closed R-49 R-982 R-985 R-991 R-1204 R-1205 R-1225 R-1226 R-1227 R-1230 R-1234; opened R-1235 R-1236 R-1237); the break-glass sheet current (D32, one A4, HU+EN); runbooks for D33/D34; poster facts; evidence day-2026-10-23
gates / gates (push) Successful in 12m53s
gates / gates (push) Successful in 12m53s
This commit is contained in:
@@ -16,6 +16,11 @@
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **2026-10-23 morning — the operator's „go" for the day brief: the releases, D31 as built, D32–D36 (decide rows in `STATUS-HISTORY.md`).**
|
||||
> **Releases:** controller 0.340.0 (global floor 06:41 UTC, MinAgent 0.158.1); agent 0.163.0 (binary sha256 561b5316…, bundle 3afb88d9…, vouched by the operator 07:58 UTC; delivered to the three test boxes by signed jobs through a step bundle `0.163.0-step1` — the D4 drop-in is a new path, R-880); installer 1.36.0 (tag `installer-v1.36.0` → 83388c08, both webpage pins); hub 0.167.0 (sha256:177860a9…, 08:27 UTC); the website `0bb57a4f`.
|
||||
> **D31** = as built (the re-issue reads the newest deletion of a KEY-HOLDING host; an install that got the password but no recovery package is rotated without a press). **D32** = yes: the sheet is names and places only, one A4, HU + EN; the operator prints and fills it. **D33** = yes: `scripts/release-copy/` nightly 01:40, newest 3 image tags (controller, hub, samba) + 5 agent versions, skopeo `dir:` (digest-true — why not `docker save`: a re-push re-compresses), ep0 `/srv/felhom-release-copy` under the one limited user `felhom-relcopy` (rrsync), cap 3 GB enforced by the sender, a read-only registry token of its own. **D34** = yes: hub snapshots at 02/08/14/20 (catch-up > 6 h), DooPlex push :30 after, limit 7 h + a wait for a snapshot being written (R-1234: 2 h was wrong), alarm 13 h; ep0 retention unchanged (keep-daily 14, keep-weekly 8). **D35** = yes, after the release week (R-1228, dated 2026-10-30). **D36** = yes, both sentences, published.
|
||||
> **Added on the operator's word the same day (not soaked):** D34's hub half went into 0.167.0 (asked before the build). Record: `audits/day-2026-10-23/`, `REPORT-day-2026-10-23.md`.
|
||||
|
||||
> **2026-10-22 evening — operator rulings D29, D30, D31 (CC's picks accepted; the decide rows now in `STATUS-HISTORY.md`).**
|
||||
> **D29** = no: the controller repository stays public for now (R-925). **D30** = the agent makes a new WireGuard key and
|
||||
> registers it on the existing peer path (R-1204). **D31** = the hub skips a deleted install that never made a key when it
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
# REPORT — day 2026-10-23: the releases and the DooPlex hour (the operator's „go")
|
||||
|
||||
Design read: `07-backup-architecture.md` §6.3/§8, `09-update-architecture.md` §3 decision 250, `03-host-agent.md`,
|
||||
`05-hub-architecture.md` §16.3, `runbooks/total-loss-of-dooplex.md`, `audits/dooplex-survival-2026-10-22/PLAN.md`.
|
||||
Evidence: `documentation/audits/day-2026-10-23/01`–`14`.
|
||||
|
||||
## Five lines
|
||||
|
||||
1. Released: controller 0.340.0 (global floor), agent 0.163.0 (vouched by the operator), installer 1.36.0, hub 0.167.0,
|
||||
the website with D36's two sentences.
|
||||
2. Root SSH from the LAN is key-only on all three test boxes; new installs get it from installer 1.36.0.
|
||||
3. The hub database now leaves every 6 hours (restored back byte-identical); the released images and agent packages
|
||||
leave nightly to ep0 (pulled back, every sha256 OK, one image loaded on a scratch box).
|
||||
4. One false failure mail at 10:28 (my install of the 6-hourly copy) — filed R-1234, fixed and red-proved the same hour.
|
||||
5. Register 191 → 184; the operator: print the sheet, Hosts page clean-up, D37 + D38.
|
||||
|
||||
## Per box — versions before → after, and the SSH checks
|
||||
|
||||
| Box | controller | agent | root-file bundle | `sshd -T` root | sshd reloaded | key login | felhom-sshd | swap = RAM (9201) |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| Tester 1 | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 (via 0.163.0-step1) | without-password | yes, 09:27:38 | ok | active, `sshd -t` ok | 4096/4096 |
|
||||
| demo-hp | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 | without-password | yes, 09:19:44 | ok | active, `sshd -t` ok | 25898/25898 |
|
||||
| demo-felhom | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 | without-password | yes, 09:24:42 | ok | active, `sshd -t` ok | 12288/12288 |
|
||||
| hub | 0.165.0 → 0.167.0 | | | | | | | |
|
||||
|
||||
Not proven (said plainly): the real root password over SSH (CC does not hold it — a dummy one is refused, but OpenSSH logs
|
||||
only „Failed password"); a felhom-sshd connection (it admits only the operator's tunnel peer); the physical console (the
|
||||
drop-in does not touch it; `passwd -S root` = P).
|
||||
|
||||
## Digests, sha256, CI by commit
|
||||
|
||||
| What | Value |
|
||||
|---|---|
|
||||
| controller 0.340.0 | `sha256:a8d1c6458da4ea9d667f928498656a291771e0b3756b20f2fcc8c1cf9d516621` — `caaa86e`, CI 2245 success |
|
||||
| agent 0.163.0 binary / bundle | `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c` / `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8` — `1109092` CI 2246, `b4721ee` CI 2249 success |
|
||||
| step bundle 0.163.0-step1 | `6cbc066902d5f3294765d5c7dc7113166c1dca25e994dbc36bfc5f139bb61fcf` |
|
||||
| installer 1.36.0 (served = tag) | `de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec` — `83388c08` CI 2247 success; pins `2aebf1bd` |
|
||||
| hub 0.167.0 | `sha256:177860a996cb285d935777d6ffdf85725f6cf4489885f54fd3bf8135c68041a0` — built at `d306b8c2` (hub/ = `be7e97dc`, CI 2250 success; `d306b8c2` CI 2251 success); manifest `34979d5d` |
|
||||
| website | `0bb57a4f` |
|
||||
| D34 fix (R-1234) | `52af38df` |
|
||||
| prometheus rules | homelab-manifests `1bbef892` |
|
||||
|
||||
CI of the last commits (`2aebf1bd`, `0bb57a4f`, `34979d5d`, `52af38df` and this close-out): see the final message.
|
||||
|
||||
## Parts
|
||||
|
||||
- **1 — controller 0.340.0:** 9202 smoke, per-customer floors, three boxes green, box walk + https-only rc 0, global floor
|
||||
06:41 UTC (MinAgent 0.158.1), read back with the WAL (`01`–`06`).
|
||||
- **2 — agent + installer:** release, a step bundle because the D4 drop-in is a new path (R-880), three signed jobs per
|
||||
box (`07`); the operator's vouch read back from the DB and the live page (`10`); installer tag, both pins, a
|
||||
webpage-only sync, the served script = the tag (`11`).
|
||||
- **3 — hub 0.167.0:** built from a clean clone; on the operator's word it also carries D34's 6-hourly snapshot (asked
|
||||
before the build; not soaked); hub-only sync; 30-minute watch: 0 hub events, 0 errors, all boxes reporting, no hub mail
|
||||
(`14`).
|
||||
- **4 — website:** D3/D5 patch + D36 HU + EN; site gates; published; read back live: new text present, old gone (control:
|
||||
the previous commit), screenshots at 390 and 1366 px, no sideways scroll (`12-website-readback/`).
|
||||
- **5 — D34:** retention unchanged, server-side on ep0 (`prune-operator-hubdb` 03:45, keep-daily 14, keep-weekly 8 — the
|
||||
current day's up to 4 copies until the next prune). Hand run 501.6 MiB / 24.5 MiB uploaded in 4 s; the newest copy
|
||||
restored from ep0 with the read-only token = the pod's snapshot by sha256 `a9ff4c8a…`, integrity ok, 4 hosts, holds
|
||||
today's vouch. Alarm 13 h. Runbook updated (`13`).
|
||||
- **6 — D33:** ep0 before: `/` 14 GB free (the data disk's root IS the PBS datastore — not used). One user
|
||||
`felhom-relcopy` (locked, rrsync-confined: shell and `..` refused), one directory `/srv/felhom-release-copy`. Cap 3 GB,
|
||||
enforced by the sender (ep0 has no quota). **skopeo, not `docker save`:** `dir:` keeps the registry's bytes, so a push
|
||||
back keeps the digest; no Docker daemon. A read-only registry token of its own, minted in the Gitea pod straight into a
|
||||
0600 file. First run 676 MB, 91 s; pulled back: 141 files OK, 9 images re-verified (digests match independent records),
|
||||
controller 0.340.0 loaded on 9202 (ID = config digest). Alarm `FelhomReleaseCopyStale`. Restore steps in
|
||||
`total-loss-of-dooplex.md` (`08`).
|
||||
- **7 — D32:** the sheet is names and places only, HU + EN, one A4 (rendered: 1 page; a copy in the operator's home
|
||||
folder); every place checked to exist. demo-hp: three superseded ISOs removed by name, `/` 8.5 → 14 GB free (`09`).
|
||||
R-548: the box half done, the product fix proposed (R-921's pattern).
|
||||
- **8 — close-out:** CONTEXT rulings, STATUS (decide rows D32–D36 → history, D37/D38 new, the do list, D35 dated
|
||||
2026-10-30), REPORTs and CHANGELOGs; the system poster's facts file updated, its hub-DB label edited (text), and a new
|
||||
arrow asked for in STATUS („System poster needs a refresh").
|
||||
|
||||
## Register — 191 → 184; opened 4, closed 11
|
||||
|
||||
Opened: R-1234 (closed the same hour), R-1235, R-1236, R-1237. Closed: R-49, R-982, R-985, R-991, R-1204, R-1205,
|
||||
R-1225, R-1226, R-1227, R-1230, R-1234. Updated: R-1214 (released; dated check 2026-10-24), R-1187, R-1189, R-232,
|
||||
R-548, R-923, R-924, R-1228 (D35, dated 2026-10-30), R-1183's check text (four snapshot slots).
|
||||
|
||||
## Observations
|
||||
|
||||
- The D34 first push refused the hub's own catch-up and mailed admin@ — FILED: R-1234 (closed, fixed `52af38df`).
|
||||
- The release copy is proven to load, not to restore a registry; it follows the newest versions, not the floor/vouch; no
|
||||
golden — FILED: R-1235.
|
||||
- Gitea OOM-killed at its 1 GiB limit, the last at 06:41:10 UTC during the release — FILED: R-1236 (STATUS D37).
|
||||
- The night sweep's mail: one-time test-box codes in one session record — FILED: R-1237.
|
||||
- The hub DB snapshot is 501 MiB, not 24 MB — NOT-A-FINDING: 24 MB is the compressed upload, which is what ep0 stores.
|
||||
- felhom-sshd does not answer from the LAN — NOT-A-FINDING: its firewall admits only the operator peer (H1, by design).
|
||||
- A dummy root password logs „Failed password", not a policy refusal — NOT-A-FINDING: OpenSSH's behaviour under
|
||||
`prohibit-password`; the policy is proven by `sshd -T` and the reload.
|
||||
- The vouch left `artifact_wrapper_sha256` empty — NOT-A-FINDING: an optional field; the wrapper rides the vouched bundle.
|
||||
- Guests 9202, 9401 (demo-hp) and a transient 990000 (Tester 1) have swap ≠ RAM — NOT-A-FINDING: outside the felhom pool.
|
||||
- `known_hosts` holds an older key for 192.168.0.155 (demo-hp's guest address) — NOT-A-FINDING: guests are reached by
|
||||
`pct exec`; nothing was changed.
|
||||
- The D4 drop-in needed a step bundle — NOT-A-FINDING: the documented R-880 path for a new bundle path.
|
||||
|
||||
## Teardown — three layers
|
||||
|
||||
Machine: 9202 — the test image untagged, the tar removed. Host: demo-hp `/tmp` tar removed; three ISOs removed on purpose
|
||||
(Part 7); DooPlex: the read-back copies and the hub-DB scratch restore shredded/removed. Hub: nothing provisioned; the
|
||||
per-customer floors cleared after the global floor. Left on purpose: the ep0 user + directory, the DooPlex release-copy
|
||||
job, the 6-hourly hub-DB timer, both alarms.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
|
||||
|
||||
**Updated 2026-10-23 (morning).** Your three picks are built (the tunnel repairs itself with a new key, the hub skips a keyless deleted install, the controller repository stays public), and the night copy can no longer outlive the controller. Four fresh reviews found 34 points, every real one fixed or a row; the test night PASSED (no failed check at night, every night job passed, nothing lost). Nothing was released or deployed; the morning list has the order, and the DooPlex survey asks four questions (D32–D35).
|
||||
**Updated 2026-10-23 (day).** Released on your go: controller 0.340.0 (the floor), agent 0.163.0 (vouched; root SSH by key only on the three test boxes), installer 1.36.0, hub 0.167.0, and the website with your two sentences. The hub database now leaves every 6 hours and the released images and agent packages every night, both read back from ep0; the break-glass sheet is current — please print it. One false failure mail reached you at 10:28 (my install of the 6-hour copy, fixed in the same hour).
|
||||
|
||||
## What works
|
||||
- A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4).
|
||||
@@ -21,7 +21,7 @@
|
||||
- The scratch box cannot use agent features (R-1083).
|
||||
|
||||
## What we work on
|
||||
- Dated checks, run by CC. 2026-10-23: no app stop when the host is busy (R-1173). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts).
|
||||
- Dated checks, run by CC. 2026-10-24: the 8 GB box's first night copy on 0.340.0 (R-1214). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes).
|
||||
|
||||
## Waiting on you — decide
|
||||
|
||||
@@ -29,11 +29,8 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
|
||||
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|
||||
|---|---|---|---|---|---|---|
|
||||
| D36 | R-1225, R-1226 | Two website sentences before the publish: „8 GB runs the basics; the photo app needs 16 GB” and „what you open from outside the home passes Cloudflare's network, readable there; the backups leave only encrypted”? | **Yes, both** (the patch's own aim is the whole truth) | — | the patch publishes with both promises a little too strong | — |
|
||||
| D35 | R-1228 | After the release week, serve the website and the installer from Cloudflare, like the ISO? | **Yes, later** (no new installs stop when DooPlex is down) | keep them home: after a DooPlex loss, no install until it is back | the same | — |
|
||||
| D34 | R-232 | Copy the hub database to ep0 every 6 hours instead of every night? | **Yes** (0 €, about 30 minutes, one DooPlex change) | keep nightly: up to a day of hub changes lost, e.g. a new box's recovery package | the same | — |
|
||||
| D33 | R-1227 | Copy the released images and agent packages to ep0 every night? | **Yes** (0 €, about 2 hours of CC, a few GB on ep0) | a timed rebuild drill instead: about 3 hours once, and rebuilt images are not the same | after a DooPlex loss no box updates, reinstalls or installs until everything is rebuilt | — |
|
||||
| D32 | R-923, R-924 | Print the break-glass sheet now (eight keys and logins + ep0's fingerprint) and keep it outside the home? | **Yes** (30 minutes; nothing else opens the off-site copies after a loss) | an encrypted USB stick outside the home: ~10 €, 1 hour, one more passphrase | after a DooPlex loss no off-site copy opens | — |
|
||||
| D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — |
|
||||
| D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — |
|
||||
| D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
|
||||
| D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — |
|
||||
|
||||
@@ -48,17 +45,14 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14.
|
||||
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
|
||||
|
||||
**Morning, 2026-10-23 — in this order.**
|
||||
1. Hold or go: CC says go — the test night passed on a fresh 8 GB box (no failed check at night, every night job passed, nothing lost; one gap: no whole-box copy to ep0, R-1232). One question from the review: D31 as built also covers an install that got the whole-box password but no recovery package — its password is rotated without your press (R-1205). CC: go. If not: revert one hub commit before item 4.
|
||||
2. Release controller 0.340.0 (main after tonight; tested as rc.8 on the scratch box, rc.6 on the test box): the safe night copy, the catch-up fix, R-49, R-1187, R-1189, R-1214. If not: these wait.
|
||||
3. Release the agent (the tunnel repair, its tunnel field, the root-SSH bundle) and the installer tag with the root-SSH removal (R-1204, R-982); then vouch. If not: a clock-ahead box stays cut off; root keeps its password on the LAN.
|
||||
4. Build and deploy hub 0.167.0 (D31, the tunnel alarm and its clearing). If not: a dead tunnel stays silent.
|
||||
5. Publish the website patch (D3, D5), with D36's two sentences if yes. If not: the old wording stays.
|
||||
6. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-16fdef (tonight's test box). If not: two dead entries alarm.
|
||||
7. The DooPlex hour: D32–D35 in the table. If not: a DooPlex loss stays as the survey describes.
|
||||
**Day, 2026-10-23 — your do list.**
|
||||
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
|
||||
2. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-16fdef. If not: two dead entries alarm.
|
||||
3. D37 and D38 in the table (two small yes/no). If not: Gitea may restart in release hours; no golden in the release copy.
|
||||
**The rest**
|
||||
- System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the hub-database label already says every 6 hours. If not: the drawing misses one copy.
|
||||
- Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12.
|
||||
- Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09.
|
||||
- Two leftovers on DooPlex: the Facebook `first-post/` folder in `leftovers/2026-10-14/` (keep, commit or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). If not: they stay. Since 2026-10-14.
|
||||
- Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784); print the break-glass sheet and the signing keys (R-923, R-924).
|
||||
- Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784);.
|
||||
- Later (yours, not urgent): R-1229 (ArgoCD's sync settings into git), R-862 (Tester 2's bootstrap, when it is back), R-504, R-1071, R-1073, R-1077, R-1042, R-132, R-908, R-779, R-904, R-232, R-433, R-933, R-213, R-832, R-388, R-882, R-883, R-814, R-1056, R-1164, R-930, R-920, R-902.
|
||||
|
||||
@@ -101,6 +101,7 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
|
||||
- one namespace **per household** (whole-guest copies, each household's own key),
|
||||
with a NEUTRAL note under it (not a red gap box): "By design: each household holds the key to its own copies; the operator cannot open them without their recovery code." — *fix (c), 2026-10-09: this is a property of the design, not a gap, and a red box said the opposite of what it means.*
|
||||
- the **`operator`** namespace: two groups — **"hub database"** and **"DooPlex copy"** (code, password manager, secrets, signing keys).
|
||||
- outside PBS, a plain folder: the **release copy** (images + agent packages, D33).
|
||||
Prune jobs on ep0: households keep one copy per week for 2 weeks (`keep-weekly 2`, R-1162, 2026-10-19; every new household's job is made with its namespace, R-1165); `operator` keeps 14 daily + 8 weekly.
|
||||
Mark ep0 with a small shield: "**Protected** — holds the only off-premises copy of real household data."
|
||||
- **Hetzner Storage Box** (grey): the shared box for the households' per-app restic copies (Tier 3), one sub-account per household, add-only keys.
|
||||
@@ -121,8 +122,9 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
|
||||
|
||||
Arrows from DooPlex (dashed, orange → ep0):
|
||||
- **00:20 "DooPlex copy"**: the code (614 MB), Gitea's database and settings, the **password manager**, the nightly secrets export, the **signing keys**. Encrypted on DooPlex with the **DooPlex off-site key**. The key that writes it **cannot delete** old copies. Tested back every Sunday.
|
||||
- **02:30 "Hub database"**: encrypted with the **hub-DB key**. Tested back every Sunday at 04:30.
|
||||
- **Left out on purpose:** the container registry (27.7 GB) — "rebuilt from the code".
|
||||
- **02:30, 08:30, 14:30, 20:30 "Hub database"** (every 6 hours since 2026-10-23, D34): encrypted with the **hub-DB key**. Tested back every Sunday at 04:30.
|
||||
- **01:40 "Release copy"** (since 2026-10-23, D33): the newest 3 released images of the controller, hub and samba, and the newest 5 agent packages — not encrypted (nothing secret), into a plain folder on ep0 (`/srv/felhom-release-copy`, its own limited user, outside PBS), about 0.7 GB, capped at 3 GB.
|
||||
- **Left out on purpose:** the rest of the container registry (older tags, test builds, non-Felhom images) — "rebuilt from the code".
|
||||
Arrow ep0 → DooPlex (dashed, 05:00): "**ep0-copy** pull: a second copy of ep0 at home. Keeps 8 weekly copies. A daily job removes a deleted customer's copy within 30 days."
|
||||
|
||||
**The test machines** (small, plain tiles near DooPlex, labelled "disposable"):
|
||||
|
||||
@@ -133,7 +133,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
|
||||
<div style="position: absolute; left: 1720px; top: 706px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: center;">The hub never connects into a box.</div>
|
||||
<div style="position: absolute; left: 1720px; top: 784px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">00:20 DooPlex copy</div>
|
||||
<div style="position: absolute; left: 1720px; top: 824px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #a85400; text-align: center;">DooPlex off-site key</div>
|
||||
<div style="position: absolute; left: 1720px; top: 872px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">02:30 hub database</div>
|
||||
<div style="position: absolute; left: 1720px; top: 872px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">every 6 h: hub database</div>
|
||||
<div style="position: absolute; left: 1720px; top: 912px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #a85400; text-align: center;">hub-DB key</div>
|
||||
<div style="position: absolute; left: 1720px; top: 962px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #0a3d73; text-align: center;">05:00 ep0-copy pull</div>
|
||||
<div style="position: absolute; left: 1720px; top: 1002px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #0a3d73; text-align: center;">ciphertext only</div>
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
== 9202 smoke 06:37:25
|
||||
switched to gitea.dooplex.hu/admin/felhom-controller:0.340.0 at 06:37:28
|
||||
healthy after ~10s: Up 7 seconds (healthy)
|
||||
2026/10/23 06:37:28 [INFO] [tier2] start-up: no leftover second-drive copy container
|
||||
errors, unhealthy+restarting: 0 0
|
||||
login page: 200
|
||||
@@ -0,0 +1,3 @@
|
||||
tester-1: HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set
|
||||
demo-hp: HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set
|
||||
demo-felhom: HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set
|
||||
@@ -0,0 +1,15 @@
|
||||
### root@192.168.0.154
|
||||
containers=10 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.340.0 Up 53 seconds (healthy)
|
||||
errors=0
|
||||
copy-containers=0
|
||||
2026/10/23 06:38:21 [INFO] [tier2] start-up: no leftover second-drive copy container
|
||||
### hp
|
||||
containers=24 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.340.0 Up 54 seconds (healthy)
|
||||
errors=0
|
||||
copy-containers=0
|
||||
2026/10/23 06:38:22 [INFO] [tier2] start-up: no leftover second-drive copy container
|
||||
### felhom-pve
|
||||
containers=9 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.340.0 Up 58 seconds (healthy)
|
||||
errors=0
|
||||
copy-containers=0
|
||||
2026/10/23 06:38:05 [INFO] [tier2] start-up: no leftover second-drive copy container
|
||||
@@ -0,0 +1,33 @@
|
||||
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
|
||||
== tester-1 (enkicsifelhom.hu): 6 public name(s)
|
||||
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
|
||||
== demo-hp (enkisfelhom.hu): 14 public name(s)
|
||||
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
|
||||
== demo-felhom (demo-felhom.eu): 8 public name(s)
|
||||
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
|
||||
== every public name redirects http -> https
|
||||
@@ -0,0 +1,13 @@
|
||||
== tester-1 (dashboard login with the household password, the press a browser makes)
|
||||
python3: can't open file '/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/38fde0c6-e964-4ec1-b0c7-56e62d675225/scratchpad/n23/fileswalk.py': [Errno 2] No such file or directory
|
||||
== demo-hp (no session): GET /__files/open -> HTTP/2 302 location: /login
|
||||
== demo-felhom (no session): GET /__files/open -> HTTP/2 302 location: /login
|
||||
== tester-1 06:40:35
|
||||
Fájlok press: GET /__files/open -> ['HTTP/2 302'] https://files.enkicsifelhom.hu/_felhom/sso
|
||||
manual second-drive copy: 200 2. mentés elindítva
|
||||
2026/10/23 06:40:43 [INFO] [backup] Tier 2 copied bookstack → /mnt/felhom-drives/adatlemez/backups/secondary/bookstack (157.0 MB, 0 leg(s), 1s)
|
||||
2026/10/23 06:40:46 [INFO] [backup] Tier 2 copied paperless-ngx → /mnt/sys_drive/felhom-data/backups/secondary/paperless-ngx (85.9 MB, 1 leg(s), 2s) [SSD: state-only]
|
||||
2026/10/23 06:40:48 [INFO] [backup] Tier 2 copied privatebin → /mnt/felhom-drives/adatlemez/backups/secondary/privatebin (16.4 KB, 0 leg(s), 0s)
|
||||
2026/10/23 06:40:48 [INFO] [backup] Tier 2 run complete: 3 app(s) processed (incl. volume-only — F6)
|
||||
copy-containers-left=0
|
||||
[exited with code 0]
|
||||
@@ -0,0 +1,5 @@
|
||||
== global floor 06:41:04: HTTP/1.1 303 See Other Location: /configuration?flash=floor_set
|
||||
clear tester-1: HTTP/1.1 303 See Other Location: /customers/tester-1?flash=floor_set
|
||||
clear demo-hp: HTTP/1.1 303 See Other Location: /customers/demo-hp?flash=floor_set
|
||||
clear demo-felhom: HTTP/1.1 303 See Other Location: /customers/demo-felhom?flash=floor_set
|
||||
read back (with WAL): [('min_controller_version', '0.340.0', '2026-10-23 06:41:04'), ('min_controller_version_declared_min_agent', '0.340.0=0.158.1', '2026-10-23 06:41:04')]
|
||||
@@ -0,0 +1,78 @@
|
||||
# Part 2 — agent 0.163.0 (operator go 2026-10-23), read 2026-10-23T07:28:53Z
|
||||
|
||||
## release (scripts/release-agent.sh at felhom-agent 1109092, CI 2246 green; CHANGELOG b4721ee CI 2249 green)
|
||||
[release-agent] built ok: sha256 561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c
|
||||
[release-agent] config bundle built: sha256 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8
|
||||
[publish-agent] upload OK (HTTP 201)
|
||||
[publish-agent] round-trip GET verified (sha256 matches)
|
||||
[release-agent] config bundle published (HTTP 201)
|
||||
sha256 : 561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c
|
||||
bundle : 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 (felhom-config-bundle.json — vouch it with the agent)
|
||||
* [new tag] v0.163.0 -> v0.163.0
|
||||
|
||||
## step bundle (R-880: the D4 drop-in is a NEW path; the boxes ran bundle 0.160.0 = 6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f)
|
||||
0.163.0-step1 = base 0.160.0 with only /usr/local/sbin/felhom-os-apply replaced; sha256 6cbc066902d5f3294765d5c7dc7113166c1dca25e994dbc36bfc5f139bb61fcf; PUT 201; read back by download: same sha
|
||||
|
||||
## delivery: three signed jobs per box (felhom-opsign, key felhom-op-1): agent_update 06:48Z, agent_config_update 0.163.0-step1 06:59Z, agent_config_update 0.163.0 07:19Z
|
||||
|
||||
## hp (192.168.0.104) 2026-10-23T07:26:29Z
|
||||
agent: felhom-agent 0.163.0
|
||||
bundle: 0.163.0 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 2026-10-23T07:19:44Z
|
||||
1 BUNDLE DONE agent=0.163.0 written=1 same=30 self-check=ok
|
||||
2 BUNDLE DONE agent=0.163.0-step1 written=1 same=29 self-check=ok
|
||||
guest 9201: memory: 25898 swap: 25898
|
||||
guest 9202: memory: 25898 swap: 512
|
||||
guest 9401: memory: 10240 swap: 0
|
||||
drop-in: -rw-r--r-- 1 root root 690 Oct 23 09:19 /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf
|
||||
drop-in content: PermitRootLogin prohibit-password
|
||||
sshd -T: permitrootlogin without-password
|
||||
root password set (passwd -S): P
|
||||
felhom-sshd: active; its root policy: permitrootlogin without-password
|
||||
felhom-sshd listens: 0.0.0.0:8822 [::]:8822 config test: ok; firewall admits only: elements = { 10.77.0.250 }
|
||||
main sshd reloaded after the drop-in: Oct 23 09:19:44 demo-hp sshd[1010]: Received SIGHUP; restarting. Oct 23 09:19:44 demo-hp systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server.
|
||||
pveproxy: active
|
||||
main sshd banner from DooPlex (:22): SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
|
||||
key login: ok
|
||||
root password attempt rc=255 (non-zero = refused)
|
||||
Oct 23 09:26:37 demo-hp sshd-session[3026730]: Failed password for root from 192.168.0.180 port 34150 ssh2
|
||||
## root@192.168.0.154 (192.168.0.154) 2026-10-23T07:28:00Z
|
||||
agent: felhom-agent 0.163.0
|
||||
bundle: 0.163.0 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 2026-10-23T07:27:38Z
|
||||
2 BUNDLE DONE agent=0.163.0 written=1 same=30 self-check=ok
|
||||
2 BUNDLE DONE agent=0.163.0-step1 written=1 same=29 self-check=ok
|
||||
guest 9201: memory: 4096 swap: 4096
|
||||
guest 990000: memory: 4096 swap: 512
|
||||
drop-in: -rw-r--r-- 1 root root 690 Oct 23 09:27 /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf
|
||||
drop-in content: PermitRootLogin prohibit-password
|
||||
sshd -T: permitrootlogin without-password
|
||||
root password set (passwd -S): P
|
||||
felhom-sshd: active; its root policy: permitrootlogin without-password
|
||||
felhom-sshd listens: 0.0.0.0:8822 [::]:8822 config test: ok; firewall admits only: elements = { 10.77.0.250 }
|
||||
main sshd reloaded after the drop-in: Oct 23 09:27:38 felhom sshd[890]: Received SIGHUP; restarting. Oct 23 09:27:38 felhom systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server.
|
||||
pveproxy: active
|
||||
main sshd banner from DooPlex (:22): SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
|
||||
key login: ok
|
||||
root password attempt rc=255 (non-zero = refused)
|
||||
Oct 23 09:28:08 felhom sshd-session[4184378]: Failed password for root from 192.168.0.180 port 33442 ssh2
|
||||
## felhom-pve (192.168.0.162) 2026-10-23T07:28:11Z
|
||||
agent: felhom-agent 0.163.0
|
||||
bundle: 0.163.0 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8 2026-10-23T07:24:43Z
|
||||
2 BUNDLE DONE agent=0.163.0 written=1 same=30 self-check=ok
|
||||
2 BUNDLE DONE agent=0.163.0-step1 written=1 same=29 self-check=ok
|
||||
guest 9201: memory: 12288 swap: 12288
|
||||
drop-in: -rw-r--r-- 1 root root 690 Oct 23 09:24 /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf
|
||||
drop-in content: PermitRootLogin prohibit-password
|
||||
sshd -T: permitrootlogin without-password
|
||||
root password set (passwd -S): P
|
||||
felhom-sshd: active; its root policy: permitrootlogin without-password
|
||||
felhom-sshd listens: 0.0.0.0:8822 [::]:8822 config test: ok; firewall admits only: elements = { 10.77.0.250 }
|
||||
main sshd reloaded after the drop-in: Oct 23 09:24:42 demo-felhom sshd[1011]: Received SIGHUP; restarting. Oct 23 09:24:42 demo-felhom systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server
|
||||
pveproxy: active
|
||||
main sshd banner from DooPlex (:22): SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
|
||||
key login: ok
|
||||
root password attempt rc=255 (non-zero = refused)
|
||||
|
||||
## notes
|
||||
- felhom-sshd :8822 admits only the operator peer 10.77.0.250 over wg-felhom (H1); a connection test from DooPlex times out BY DESIGN — proven: active, listening, sshd -t ok, firewall set.
|
||||
- a dummy root password is logged as 'Failed password' (OpenSSH refuses a root password silently under prohibit-password); the policy proof is sshd -T + the SIGHUP reload after the drop-in. The real root password was not tried (CC does not hold it).
|
||||
- guests 9202, 9401 (demo-hp) and the transient 990000 (Tester 1, gone a minute later) are outside the felhom pool; swap = RAM applies to pool guests (9201 on all three: swap = memory).
|
||||
@@ -0,0 +1,53 @@
|
||||
# Part 6 — D33: the released images and agent packages to ep0 nightly (operator yes 2026-10-23), 2026-10-23T07:51:56Z
|
||||
|
||||
## ep0 — free space first, then ONE user and ONE directory (nothing else on ep0 changed)
|
||||
before: /dev/sda1 38G 23G used 14G free 63% (/mnt/pbs-datastore is the PBS datastore root — not used)
|
||||
created: user felhom-relcopy (system, uid 999, password locked 'L', shell /bin/sh for the forced command), /srv/felhom-release-copy (0750) with data/ and .ssh/authorized_keys:
|
||||
command="/usr/bin/rrsync /srv/felhom-release-copy/data",restrict ssh-ed25519 <the job key, felhom-release-copy@dooplex>
|
||||
fence test: rsync --list-only OK; a shell command → 'rrsync error: SSH_ORIGINAL_COMMAND does not run rsync'; '..' → 'do not use .. in arg'
|
||||
after the first copy: /dev/sda1 38G 24G used 13G free 65%; /srv/felhom-release-copy/data 677M
|
||||
cap: CAP_MB=3072, enforced by the sender (ep0's root fs has no quota; a quota would change more than one directory)
|
||||
|
||||
## DooPlex — the job (felhom.eu d306b8c2): /usr/local/sbin/felhom-release-copy + .service + .timer (01:40), /etc/felhom-release-copy/{env,id_ed25519,known_hosts,registry-token}
|
||||
registry token: a NEW Gitea access token for user admin, name felhom-release-copy-d33, scope read:package ONLY — minted inside the gitea pod straight into /etc/felhom-release-copy/registry-token (0600 root); never printed
|
||||
known_hosts: ep0's ed25519 host key, SHA256:NGtcS5v3K+9UjMTM6X5Ib0CoChdhqj9bezLz1hOUqgY — equal to the key already pinned for root@ep0
|
||||
skopeo, not docker save: skopeo dir: keeps the registry's manifest bytes and compressed blobs, so a push back gives the SAME digest (docker save re-compresses → a new digest); no Docker daemon needed
|
||||
|
||||
## the first run (systemctl start, under the unit's sandbox)
|
||||
2026-10-23T09:48:33+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-controller: keeping 0.338.0, 0.339.0, 0.340.0
|
||||
2026-10-23T09:48:37+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-hub: keeping 0.164.0, 0.165.0, 0.166.0
|
||||
2026-10-23T09:48:43+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-samba: keeping 1.0.0, 1.1.0, 1.2.0
|
||||
2026-10-23T09:48:47+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: felhom-agent: keeping 0.159.2, 0.160.0, 0.161.0, 0.162.0, 0.163.0
|
||||
2026-10-23T09:48:51+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: staged and verified: 676 MB, new this run: felhom-controller:0.338.0, felhom-controller:0.339.0, felhom-controller:0.340.0, felhom-hub:0.164.0, felhom-hub:0.165.0, felhom-hub:0.166.0, felhom-samba:1.0.0, felhom-samba:1.1.0, felhom-samba:1.2.0
|
||||
2026-10-23T09:49:51+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: pushed to ep0 and compared by checksum: no difference (91 s)
|
||||
2026-10-23T09:49:51+02:00 dooplex felhom-release-copy[1567269]: felhom-release-copy: success signal written
|
||||
2026-10-23T09:49:51+02:00 dooplex systemd[1]: felhom-release-copy.service: Consumed 30.416s CPU time, 743.1M memory peak.
|
||||
|
||||
## read back from ep0 (rsync pull with the job key → /var/lib/felhom-release-copy/readback)
|
||||
sha256sum -c SHA256SUMS: every file OK (141 files), 677 MB
|
||||
every image re-verified (top manifest = its digest, every blob = its name):
|
||||
felhom-controller 0.338.0 sha256:e8a529f395bfe8c0e7b87d5d98929a6f8e9834aa13c0ab77bd214ee9e7e53433
|
||||
felhom-controller 0.339.0 sha256:fee71aafceb4a7d5f24ae4a6a273a8d51eb26e0550fe9d12d4d86955f6a343a8
|
||||
felhom-controller 0.340.0 sha256:a8d1c6458da4ea9d667f928498656a291771e0b3756b20f2fcc8c1cf9d516621
|
||||
felhom-hub 0.164.0 sha256:931594e333a4e8a9784eff2fb144d022ab46f89dd6a865a93e6100dc28dbd9dc
|
||||
felhom-hub 0.165.0 sha256:c6180fb79a2ff7d5546a704c68b634b9207afb0a0ecd524c74b0310434532537
|
||||
felhom-hub 0.166.0 sha256:829667bb289c790bc99719e88dad8f7d08faaa9f662df1e9a06b1774915a1af6
|
||||
felhom-samba 1.0.0 sha256:0fa9fae3eaa9d48836c855a4884f0751a90ff050dc9e7d17e41d93bb434499c0
|
||||
felhom-samba 1.1.0 sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10
|
||||
felhom-samba 1.2.0 sha256:da96f25033827fbb77d0ee398e86fe5b744e2f29be9de898994533340c97acf9
|
||||
agent 0.159.2 binary 4d98868e922d6c10665a6c0da9323e8627ea9c589094e81d27d741e625d9d8f9 bundle 477704ea2980948433ffc34bb59f008cc11410fd9683d386bfd3e2f085d55bc9
|
||||
agent 0.160.0 binary 7ebb547f2b1ab93544ba828d3040351b60c4126327a5b85bbcb33f8f2b3dfca1 bundle 6b06090190ff609af1fe9842447ebf385fd52da4be382bc7ca708b558c8e202f
|
||||
agent 0.161.0 binary ff4e6dc52dd70234913445ecdeb88d31dea8e394d77c48704d26ff19b999d3b5 bundle 485f36c957d83f772aa22cac24e5e73f44da5cf8b957b1fd513bec2bc1d12c47
|
||||
agent 0.162.0 binary 5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928 bundle ac772967fd4669045e837a70bf5acca239c58d55f7f00212e472a8facd60b5a5
|
||||
agent 0.163.0 binary 561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c bundle 3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8
|
||||
independent controls (other channels): controller 0.340.0 = the release digest from build.sh (03-release-0340-boxes.txt); hub 0.165.0 = the running hub pod's imageID; hub 0.164.0 = hub/CHANGELOG; agent 0.163.0 = release-agent.sh today; agent 0.161.0 = night2-2026-10-20/P1/1b; bundle 0.160.0 = the three boxes' installed bundle_sha256
|
||||
|
||||
## load on a scratch machine (9202 on demo-hp, Tier 0)
|
||||
the ep0 copy of felhom-controller 0.340.0 → skopeo docker-archive (amd64) → pct push 9202 → docker load:
|
||||
Loaded image: felhom-restorecopy-test:0.340.0; ID=sha256:3f34d74519463c338422594b70c31d9c5beabed48847134c6d26b5a95ebbae2b = the copy's amd64 config digest; a container ran (ls /)
|
||||
teardown: image untagged (0 left), the tar removed on 9202, on demo-hp /tmp and on DooPlex
|
||||
|
||||
timer: Sat 2026-10-24 01:42:16 CEST 15h - - felhom-release-copy.timer felhom-release-copy.service
|
||||
|
||||
## alarm (homelab-manifests 1bbef892): FelhomReleaseCopyStale (> 26 h or absent, warning) — promtool check: 7 rules OK; promtool test: quiet at 25 h, fires at 27 h
|
||||
synced ConfigMap mon-system/prometheus-rules only; POST /-/reload 200; /api/v1/rules: FelhomReleaseCopyStale inactive ok; metric age at read ~3235 s
|
||||
@@ -0,0 +1,10 @@
|
||||
no guest references the three
|
||||
BEFORE: /dev/mapper/pve-root 39G 29G 8.5G 77% /
|
||||
Removed volume 'local:iso/felhom-installer-1.27.1-pve9.2-1.iso'
|
||||
removed local:iso/felhom-installer-1.27.1-pve9.2-1.iso
|
||||
Removed volume 'local:iso/felhom-installer-1.28.0-pve9.2-1.iso'
|
||||
removed local:iso/felhom-installer-1.28.0-pve9.2-1.iso
|
||||
Removed volume 'local:iso/felhom-installer-1.29.0-pve9.2-1.iso'
|
||||
removed local:iso/felhom-installer-1.29.0-pve9.2-1.iso
|
||||
AFTER: /dev/mapper/pve-root 39G 24G 14G 64% /
|
||||
local:iso/felhom-installer-1.30.0-pve9.2-1.iso iso iso 1705324544
|
||||
@@ -0,0 +1,16 @@
|
||||
# The vouch read-back — agent 0.163.0 (the operator vouched on the hub, answered 'Vouched now'), read 2026-10-23T08:00:42Z
|
||||
|
||||
## channel 1: hub DB (hub.db + -wal + -shm in ONE tar from the pod), hub_settings
|
||||
artifact_agent_sha256|561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c|2026-10-23 07:58:46
|
||||
artifact_agent_version|0.163.0|2026-10-23 07:58:46
|
||||
artifact_bundle_sha256|3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8|2026-10-23 07:58:46
|
||||
artifact_golden_sha256|3881de62b0986ccbe5e55580e3f1bb9765d65b8d898f67c6339bac105244f84f|2026-10-23 07:58:46
|
||||
artifact_golden_version|0.331.0|2026-10-23 07:58:46
|
||||
artifact_min_agent|0.158.1|2026-10-23 07:58:46
|
||||
artifact_wrapper_sha256||2026-10-23 07:58:46
|
||||
|
||||
## channel 2: the running hub's operator page GET /configuration (CC's token), the selected options
|
||||
value="0.163.0" data-sha="561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c" selected>0.163.0
|
||||
value="0.331.0" data-sha="3881de62b0986ccbe5e55580e3f1bb9765d65b8d898f67c6339bac105244f84f" selected>0.331.0
|
||||
|
||||
agent 0.163.0 sha256 = the release (561b5316…); bundle sha256 = the release (3afb88d9…). Golden 0.331.0 unchanged. The floor's MinAgent stays 0.158.1 (artifact_min_agent 0.158.1; controller 0.340.0 declares 0.158.1).
|
||||
@@ -0,0 +1,13 @@
|
||||
# Installer 1.36.0 published (operator yes 2026-10-23), read 2026-10-23T08:16:46Z
|
||||
|
||||
1. bump: felhom.eu 83388c08 (SCRIPT_VERSION 1.36.0; scripts/test_hostinstall.py 40 passed; CI 2247 success)
|
||||
2. tag: installer-v1.36.0 (annotated d00f868c) -> 83388c08, pushed
|
||||
3. pins: manifests/webpage.yaml both --ref=installer-v1.36.0 (sidecar git-sync-scripts + init git-sync-scripts-init), commit 2aebf1bd;
|
||||
ArgoCD: only Deployment felhom-system/felhom-webpage was OutOfSync; synced that ONE resource at revision 2aebf1bd; rollout ok;
|
||||
live args: git-sync-scripts-init ['--ref=installer-v1.36.0'], git-sync-scripts ['--ref=installer-v1.36.0'], website git-sync ['--ref=main'] (unchanged)
|
||||
4. read back: curl https://felhom.eu/scripts/felhom-host-install.sh
|
||||
SCRIPT_VERSION="1.36.0" # the SINGLE version source (F-1): -h and the run banners follow it.
|
||||
sha256 served = de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec
|
||||
sha256 tag file = de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec
|
||||
the drop-in name 10-felhom-root-key-only appears 2 times (the uninstall's removal)
|
||||
Rollback: move the tag back to installer-v1.35.0's commit and wait ~30 s (no sync).
|
||||
@@ -0,0 +1,4 @@
|
||||
# Website publish read-back (felhom.eu 0bb57a4f, operator yes 2026-10-23)
|
||||
Method: headless Chromium (playwright 1.61.1 image) on the LIVE site; the FAQ answer opened by a click; 390x844 (phone) and 1366x900 (computer).
|
||||
No page scrolls sideways at either size (scrollWidth == viewport on all 10 shots).
|
||||
Text: see text-check.txt (new sentences present; the old 'only encrypted (the remote backup)' gone; control: HEAD~1 had it twice).
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 144 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 122 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 113 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 122 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 124 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 70 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 54 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 59 KiB |
@@ -0,0 +1,7 @@
|
||||
waited ~0 s
|
||||
NEW letoltes 'az alapokat futtatja' count 1 (want 2 on the FAQ: visible + JSON-LD; 1 on download)
|
||||
NEW en/download 'runs the basics; the photo app needs 16 GB' count 1 (want 2 on the FAQ: visible + JSON-LD; 1 on download)
|
||||
NEW gyik 'a Cloudflare h' count 4 (want 2 on the FAQ: visible + JSON-LD; 1 on download)
|
||||
NEW en/faq "passes through Cloudflare's network and is readable there" count 2 (want 2 on the FAQ: visible + JSON-LD; 1 on download)
|
||||
OLD gyik 'otthonodat (a t' live 0 (want 0); control HEAD~1 2 (want 2)
|
||||
OLD en/faq 'leaves your home only encrypted (the remote backup)' live 0 (want 0); control HEAD~1 2 (want 2)
|
||||
@@ -0,0 +1,43 @@
|
||||
# Part 5 — D34: the hub DB off-site every 6 hours (operator yes 2026-10-23), 2026-10-23T08:41:15Z
|
||||
|
||||
## what changed
|
||||
- hub 0.167.0 (deployed 08:27Z, image sha256:177860a9…): snapshots at 02:00 08:00 14:00 20:00 Budapest; start-up catch-up when the newest > 6 h. Hub log at start:
|
||||
2026/10/23 10:28:12 [INFO] db-snapshot 08:00: next run at 2026-10-24 08:00 CEST (in 21h31m48s)
|
||||
2026/10/23 10:28:12 [INFO] db-snapshot 14:00: next run at 2026-10-23 14:00 CEST (in 3h31m48s)
|
||||
2026/10/23 10:28:12 [INFO] db-snapshot 20:00: next run at 2026-10-23 20:00 CEST (in 9h31m48s)
|
||||
2026/10/23 10:28:12 [INFO] db-snapshot 02:00: next run at 2026-10-24 02:00 CEST (in 15h31m48s)
|
||||
2026/10/23 10:30:06 [INFO] db snapshot written: hub-20261023T082812Z.db (525996032 bytes, 1m54.537s); pruned 1, keeping 2
|
||||
- DooPlex felhom-hub-db-backup (felhom.eu 52af38df): timer 02:30 08:30 14:30 20:30; waits ≤ 20 min for a snapshot being written; refuses a newest snapshot older than 7 h
|
||||
- R-1234 (closed): the first version (2 h, no wait, be7e97dc) refused the timer's own catch-up at 10:28 CEST and mailed admin@ once — fixed in 52af38df
|
||||
|
||||
## retention (ep0, server-side, unchanged): prune-job prune-operator-hubdb, ns operator, 03:45, keep-daily 14, keep-weekly 8
|
||||
=> the current day's copies (up to 4) stay until the next 03:45 prune, then the newest of each day for 14 days and one per week for 8 weeks. Size per copy: 501.6 MiB raw, 24.5 MiB uploaded (compressed, dedup'd chunks).
|
||||
|
||||
## run once by hand (systemctl start felhom-hub-db-backup.service, 10:39 CEST)
|
||||
2026-10-23T10:39:33+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: snapshot hub-20261023T082812Z.db, 11 min old
|
||||
2026-10-23T10:39:45+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: checked: 525996032 bytes, integrity ok, 4 host(s)
|
||||
2026-10-23T10:39:48+02:00 dooplex felhom-hub-db-backup[2377041]: hubdb.pxar: had to backup 501.629 MiB of 501.629 MiB (compressed 24.471 MiB) in 2.47 s (average 203.437 MiB/s)
|
||||
2026-10-23T10:39:49+02:00 dooplex felhom-hub-db-backup[2377041]: Duration: 3.21s
|
||||
2026-10-23T10:39:49+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: pushed hub-20261023T082812Z.db to ep0 (ns operator) in 4 s
|
||||
2026-10-23T10:39:49+02:00 dooplex felhom-hub-db-backup[2374380]: felhom-hub-db-backup: success signal written
|
||||
|
||||
## read back: restore the NEWEST copy from ep0 with the READ-ONLY token into a scratch dir (shredded after)
|
||||
15 copies on ep0, oldest 2026-10-10T00:30:56Z
|
||||
newest copy on ep0: host/dooplex-hub/2026-10-23T08:39:45Z
|
||||
Using encryption key from '/etc/felhom-hub-backup/enc.key'..
|
||||
Fingerprint: b2:19:bf:36:3b:97:3d:6c
|
||||
restore complete (501.629 MiB processed in 5.4s, average 92.519 MiB/s)
|
||||
restored size: 525996032 bytes
|
||||
restored sha256: a9ff4c8a0d0fe34da72411d9c165e573069f5601492d1e625578464f56adbde7
|
||||
integrity_check: ok
|
||||
SELECT COUNT(*) FROM hosts: 4
|
||||
SELECT value FROM hub_settings WHERE key='artifact_agent_version': 0.163.0
|
||||
pod snapshot hub-20261023T082812Z.db: 525996032 bytes, sha256 a9ff4c8a0d0fe34da72411d9c165e573069f5601492d1e625578464f56adbde7 (kubectl exec sha256sum — a different channel from the restore)
|
||||
=> byte-identical; it holds today's vouch (artifact_agent_version 0.163.0, set 07:58Z) — a nightly copy could not
|
||||
|
||||
## timers
|
||||
Fri 2026-10-23 14:30:08 CEST 3h 48min Fri 2026-10-23 10:28:30 CEST 12min ago felhom-hub-db-backup.timer felhom-hub-db-backup.service
|
||||
|
||||
## alarm (homelab-manifests 1bbef892): HubDBBackupStale 26 h -> 13 h (two missed 6-hourly pushes)
|
||||
promtool test: quiet at 12 h, fires at 14 h; RED-PROOF: the rule at 26 h fails the 14 h case
|
||||
in force: /api/v1/rules shows the 13 * 3600 expression, inactive ok; metric age at read ~237 s
|
||||
@@ -0,0 +1,21 @@
|
||||
# Part 3 — hub 0.167.0 deploy and the 30-minute watch (operator yes 2026-10-23), read 2026-10-23T08:56:31Z
|
||||
|
||||
build: clean clone at d306b8c2 (hub/ unchanged since be7e97dc, CI 2250 success; d306b8c2 CI 2251 success before the deploy); hub tests rc 0 (28 packages)
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.167.0 sha256:177860a996cb285d935777d6ffdf85725f6cf4489885f54fd3bf8135c68041a0
|
||||
manifest 34979d5d; ArgoCD: only Deployment felhom-system/hub OutOfSync; synced that one resource; rollout ok 08:27:34Z; pod imageID = the digest above; Synced/Healthy
|
||||
|
||||
## watch 08:27:34Z → 08:56:31Z — channel 1: the hub DB (with WAL) + log
|
||||
events since the deploy: 0 (newest event 07:07:10Z Tester4 host_down — the expected dead test box; 161 events since 2026-10-22: the table is live)
|
||||
host report|demo-felhom-8363b5|2026-10-23 08:54:54|2
|
||||
host report|demo-hp-bb76ea|2026-10-23 08:49:41|2
|
||||
host report|tester-1-d70be4|2026-10-23 08:42:35|1
|
||||
controller report|demo-felhom|2026-10-23 08:53:20|3
|
||||
controller report|demo-hp|2026-10-23 08:53:24|3
|
||||
controller report|tester-1|2026-10-23 08:53:23|3
|
||||
hub log since deploy: 0 [ERROR]; 'osupdates: alarm check done … 0 alarm(s) sent'; 'Host staleness checker initialized: 3 ok, 1 already alarmed (seeded silently, R-1001)' — Tester4 not re-mailed
|
||||
|
||||
## channel 2: the operator mailbox (Gmail connector, to:admin@felhom.eu newer_than:1d)
|
||||
after 08:27Z: ONE mail, 08:28:32Z '[DooPlex backup] FAILED: felhom-hub-db-backup.service' — NOT from the hub: CC's install of the 6-hourly timer (R-1234, closed, fixed 52af38df). No hub mail after the deploy.
|
||||
before the deploy (control that the channel delivers): 07:07:11Z Tester4 host_down, 07:04:11Z node_down, 06:22Z/06:19Z stale — the hub's mails do arrive
|
||||
|
||||
verdict: no new false alarm from the hub in the 30 minutes after the deploy; one CC-caused DooPlex mail, filed and fixed
|
||||
@@ -30,6 +30,16 @@
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|---|---|---|---|
|
||||
| **R-49** | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** (P3) | CLOSED 2026-10-23 — released in controller **0.340.0** (D21 = B: a template may exclude a named volume; Immich's model cache is out of every backup); the global floor 0.340.0 (MinAgent 0.158.1) since 06:41 UTC, all three test boxes on it | `audits/day-2026-10-23/` 03, 06 |
|
||||
| **R-1204** | **A box that runs once with its clock ahead loses its WireGuard tunnel to ep0 — the PBS whole-box copies and the operator's door — until the real date passes the wrong one, and nothing says why.** (P2) | CLOSED 2026-10-23 — the sign reported and the heal (D30, the agent re-keys on the existing peer path) in agent **0.163.0** (vouched by the operator 07:58 UTC; on Tester 1, demo-hp, demo-felhom by signed jobs), the alarm and its clearing in hub **0.167.0** (deployed 08:27 UTC). The live heal was proven on the test night (`audits/night-2026-10-23/P4-testnight/`) | `audits/day-2026-10-23/` 07, and the hub watch in 14 |
|
||||
| **R-1205** | **After a box death, a SECOND install that is deleted before it makes a key leaves the whole-box (PBS) tier waiting for the operator's re-issue.** (P2) | CLOSED 2026-10-23 — D31 in hub **0.167.0** (deployed 08:27 UTC, image sha256:177860a9…): the re-issue reads the newest deletion of a host that held a key | `audits/day-2026-10-23/` 14 |
|
||||
| **R-1230** | **A reinstalled box that waits for the household's recovery code was reported „stuck” after an hour: a false operator mail, a needless re-issue of the off-site login, and a controller restart on the box.** (P3) | CLOSED 2026-10-23 — hub **0.167.0** (deployed 08:27 UTC): the delivery checker stands down for `awaiting_recovery_key` | `audits/day-2026-10-23/` 14 |
|
||||
| **R-982** | **The Proxmox host accepts root password login over SSH from the LAN (measured on demo-hp), so anyone on the household network can try the root password.** (P3) | CLOSED 2026-10-23 — D4 delivered: the bundle drop-in in agent **0.163.0** (config bundle sha256 3afb88d9…, via a step bundle 0.163.0-step1 because it adds a path, R-880) on all three test boxes — `sshd -T` `permitrootlogin without-password`, sshd reloaded after the drop-in, a key login works, felhom-sshd active; new installs: installer **1.36.0** published (tag `installer-v1.36.0`, served sha256 de8bf5f3… = the tag). The real root password was not tried over SSH (CC does not hold it); the console is untouched by design | `audits/day-2026-10-23/` 07, 11 |
|
||||
| **R-985** | **Our zero-knowledge wording is stronger than the product: on a live box the operator can reach the off-site backup key (through the vaulted root credential and the operator tunnel) without the customer's recovery code.** (P3) | CLOSED 2026-10-23 — the D3 wording published on the website (felhom.eu `0bb57a4f`), read back live at phone and computer size | `audits/day-2026-10-23/` 12-website-readback |
|
||||
| **R-991** | **Nothing a buyer can read says which machines Felhom installs on, and a quarter of the forum beginners own only machines it cannot install on.** (P3) | CLOSED 2026-10-23 — the D5 minimum machine published (felhom.eu `0bb57a4f`), read back live | `audits/day-2026-10-23/` 12-website-readback |
|
||||
| **R-1225** | **The website's minimum machine (8 GB) cannot run the photo app, and the download page does not say so.** (P3) | CLOSED 2026-10-23 — D36: „8 GB runs the basics; the photo app needs 16 GB” on both download pages, published `0bb57a4f`, read back live | `audits/day-2026-10-23/` 12-website-readback |
|
||||
| **R-1226** | **Two FAQ answers say the household's data leaves the home only encrypted (the remote backup) — not true for data reached from outside.** (P2) | CLOSED 2026-10-23 — D36: both FAQ answers (visible + JSON-LD, HU + EN) now say what is opened from outside passes Cloudflare's network and is readable there; the backups leave only encrypted. Published `0bb57a4f`; live: the new sentence 2× per FAQ page, the old one 0× (control: the previous commit had it 2×) | `audits/day-2026-10-23/` 12-website-readback |
|
||||
| **R-1227** | **The container registry and the agent packages exist only on DooPlex — after its loss no box updates, reinstalls or installs.** (P2) | CLOSED 2026-10-23 — D33 built (felhom.eu `d306b8c2`): nightly 01:40 the newest 3 release tags of controller, hub, samba (skopeo `dir:`, digest-true) and the newest 5 agent versions to ep0 `/srv/felhom-release-copy` (one new user `felhom-relcopy`, rrsync-confined; cap 3 GB by the sender); first run 676 MB in 91 s; pulled back, every sha256 OK, every image re-verified; controller 0.340.0 from the copy loaded on 9202 (ID = config digest); alarm `FelhomReleaseCopyStale`; restore steps in `total-loss-of-dooplex.md`. Follow-ups: R-1235 | `audits/day-2026-10-23/` 08-release-copy-D33.txt |
|
||||
| **R-1234** | **D34's first push refused the hub's own catch-up and mailed a false „hub-db-backup failed" to admin@.** Installing the 6-hourly timer at 10:28 made systemd catch up the missed 08:30 slot at once (`Persistent=true`), while the freshly deployed hub 0.167.0 was still writing its start-up snapshot (`hub-*.db.tmp`); the only finished one was 02:00 (8 h), above the new 2-hour limit → „the hub stopped snapshotting", `OnFailure` mail id `01a167e0…`. The 2 h limit was also stricter than the hub's own 6-hour catch-up, so any restart could repeat it. (P3) | CLOSED 2026-10-23 day, the same session — the push waits (≤ 20 min) for a snapshot being written, and the limit is one slot gap + 1 h (7 h); tests `test_d34_waits_for_a_snapshot_being_written` (reproduces the 10:28 run when the wait is removed), `…_a_restart_within_one_slot_is_not_an_alarm`, `…_two_missed_slots_are_refused`, each red-proved | `audits/day-2026-10-23/13-hubdb-6h-D34.txt` |
|
||||
|
||||
## 2026-10-22 night — the seven decisions put into effect (full text of the rows: `git show 1100a344:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -5,6 +5,20 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
|
||||
|
||||
## Moved from STATUS on 2026-10-23 (day)
|
||||
|
||||
**Updated 2026-10-23 (morning).** Your three picks are built (the tunnel repairs itself with a new key, the hub skips a keyless deleted install, the controller repository stays public), and the night copy can no longer outlive the controller. Four fresh reviews found 34 points, every real one fixed or a row; the test night PASSED (no failed check at night, every night job passed, nothing lost). Nothing was released or deployed; the morning list has the order, and the DooPlex survey asks four questions (D32–D35).
|
||||
|
||||
**Decided 2026-10-23 morning — the operator's „go" for the day brief** (each ruling in its row and in CONTEXT.md): D32 = yes, print the sheet; D33 = yes, the nightly release copy to ep0 (built, R-1227 closed); D34 = yes, the hub DB every 6 hours (built); D35 = yes, after the release week (R-1228, dated 2026-10-30); D36 = yes, both sentences (published). The rows as they stood:
|
||||
|
||||
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|
||||
|---|---|---|---|---|---|---|
|
||||
| D36 | R-1225, R-1226 | Two website sentences before the publish: „8 GB runs the basics; the photo app needs 16 GB” and „what you open from outside the home passes Cloudflare's network, readable there; the backups leave only encrypted”? | **Yes, both** (the patch's own aim is the whole truth) | — | the patch publishes with both promises a little too strong | — |
|
||||
| D35 | R-1228 | After the release week, serve the website and the installer from Cloudflare, like the ISO? | **Yes, later** (no new installs stop when DooPlex is down) | keep them home: after a DooPlex loss, no install until it is back | the same | — |
|
||||
| D34 | R-232 | Copy the hub database to ep0 every 6 hours instead of every night? | **Yes** (0 €, about 30 minutes, one DooPlex change) | keep nightly: up to a day of hub changes lost, e.g. a new box's recovery package | the same | — |
|
||||
| D33 | R-1227 | Copy the released images and agent packages to ep0 every night? | **Yes** (0 €, about 2 hours of CC, a few GB on ep0) | a timed rebuild drill instead: about 3 hours once, and rebuilt images are not the same | after a DooPlex loss no box updates, reinstalls or installs until everything is rebuilt | — |
|
||||
| D32 | R-923, R-924 | Print the break-glass sheet now (eight keys and logins + ep0's fingerprint) and keep it outside the home? | **Yes** (30 minutes; nothing else opens the off-site copies after a loss) | an encrypted USB stick outside the home: ~10 €, 1 hour, one more passphrase | after a DooPlex loss no off-site copy opens | — |
|
||||
|
||||
## Moved from STATUS on 2026-10-23
|
||||
|
||||
**Updated 2026-10-22 (night).** Your seven picks of 15:47 are in effect or ready: D6 and D8 are live, D21 waits for the controller release, and D3, D5 (the website) and D4 (root SSH) are ready patches. Four controller fixes are built and proven on the scratch box, among them a gentler night copy (the 03:30 memory pressure) and no import over a stopped app. Nothing was released or deployed; the morning list below has the order.
|
||||
|
||||
@@ -93,6 +93,17 @@ itself. **Corrected:** one snapshot took 44 s on the live volume (0.63 s on a lo
|
||||
|
||||
### Step 4 — the push (on DooPlex, root; `felhom-hub-db-backup.service` + `.timer` 02:30) — DONE 2026-10-05
|
||||
|
||||
> **Since 2026-10-23 (D34, operator yes): every 6 hours.** The hub (≥ 0.167.0) writes a snapshot at 02:00, 08:00, 14:00
|
||||
> and 20:00 Budapest (`dbsnap.Slots`; start-up catch-up when the newest is older than 6 h); the timer pushes at 02:30,
|
||||
> 08:30, 14:30 and 20:30. The push waits (≤ 20 min) for a snapshot being written and refuses a newest snapshot older
|
||||
> than **7 h** (one slot gap + 1 h — a 2 h limit refused the hub's own catch-up once and mailed a false failure,
|
||||
> R-1234). Alarm `HubDBBackupStale` at **13 h** (two missed pushes). **Retention is unchanged and server-side on ep0**
|
||||
> (`prune-operator-hubdb`, 03:45, keep-daily 14, keep-weekly 8): the current day's copies (up to 4) live until the next
|
||||
> prune, then the newest of each day for 14 days and one per week for 8 weeks. One copy: ~500 MiB raw, ~25 MiB
|
||||
> uploaded. Proven 2026-10-23: a hand run, then the newest copy restored from ep0 with the read-only token was
|
||||
> byte-identical to the pod's snapshot, integrity ok, `SELECT COUNT(*) FROM hosts` = 4
|
||||
> (`audits/day-2026-10-23/13-hubdb-6h-D34.txt`). The text below is the original nightly design.
|
||||
|
||||
**The real script is `scripts/hub-db-backup/felhom-hub-db-backup`** (versioned, R-231; installed by `install.sh`;
|
||||
15 tests in `test_hub_db_backup.py`, run by hand — not in CI). It adds to the sketch below: it refuses a snapshot older
|
||||
than 26 h (the hub stopped snapshotting), a copy whose size differs from the pod's file, and a copy with no hosts; it
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Break-glass sheet — the keys that must exist outside DooPlex (R-923)
|
||||
# Break-glass sheet — the keys that must exist outside DooPlex (R-923, D32)
|
||||
|
||||
> **A template. It holds NO key value, and none may ever be written into this file, a commit, a log or a chat.**
|
||||
> Print this page, then write or stick each value onto the paper copy only. Keep the paper away from home (DooPlex is
|
||||
@@ -33,31 +33,43 @@ captured line on some systems; strip it when typing the value back.
|
||||
**Check a print later without exposing it:** for S1/S2 the only reliable check is one restore with a key file rebuilt
|
||||
from the printed `data` field (hub-DB runbook Step 0 note) — `key show` cannot check a rebuilt key.
|
||||
|
||||
## Public fingerprints — to check you hold the right key (not secret; not on the printed page)
|
||||
|
||||
| # | Fingerprint |
|
||||
|---|---|
|
||||
| S1 | PBS key `93:03:bf:d7:1f:4c:9e:fe…` (`proxmox-backup-client key show`) |
|
||||
| S2 | PBS key `b2:19:bf:36:3b:97:3d:6c…` |
|
||||
| S6a | `SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k` |
|
||||
| S6b | `SHA256:7YqN4rXO08yixTeOO+UtQ8jHyIGycICuctQgRYVGnWw` |
|
||||
| S19 | recipient `age1aw83ej37x9sck89xwt856ry0553kf5sng3w544an4et54yjdx5qq79utrv` |
|
||||
|
||||
Every place below was checked to exist on 2026-10-23 (existence only, no value read). The release copy on ep0 (D33)
|
||||
and the 6-hourly hub-DB copy (D34) add NO key: the release copy is not encrypted (it holds no secret), and the hub-DB
|
||||
copy uses S2 as before.
|
||||
|
||||
---
|
||||
|
||||
## The sheet (print from here)
|
||||
## The sheet (print from here — one A4 page)
|
||||
|
||||
**FELHOM — break-glass keys.** Printed on: ____________ Stored at: ______________________________
|
||||
**FELHOM — vészhelyzeti kulcsok / break-glass keys.** Nyomtatva / printed: ____________ Hol tárolom / kept at: ______________________
|
||||
|
||||
| # | Key | Public fingerprint (to match the right key) | Value — write or stick here |
|
||||
Csak nevek és helyek; az értéket kézzel írd be. *Names and places only; write each value by hand.* Tartsd az otthonon kívül. *Keep it outside the home.*
|
||||
|
||||
| # | Név / Name | Hol van most / Where it lives now | Érték / Value |
|
||||
|---|---|---|---|
|
||||
| S1 | DooPlex off-site key — opens Gitea, the password manager, the k8s Secrets export (ep0 `operator`, `host/dooplex-gitea`) | PBS key `93:03:bf:d7:1f:4c:9e:fe…` | `data`: ______________________________________________ |
|
||||
| S2 | Hub-DB off-site key — opens the hub database (ep0 `operator`, `host/dooplex-hub`) | PBS key `b2:19:bf:36:3b:97:3d:6c…` | `data`: ______________________________________________ |
|
||||
| S4 | Hub seal key `OFFSITE_SECRET_KEY` (64 hex) | — | ______________________________________________________ |
|
||||
| S5 | DooPlex restic / Secrets-export passphrase | — | ______________________________________________________ |
|
||||
| S6a | Signing RECOVERY key `felhom-rec-recovery` (also in the S1 copy since 2026-10-09; the paper is the copy that needs nothing else) | `SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k` | (staple the printout) |
|
||||
| S6b | Signing OPERATIONAL key `felhom-op-operational` | `SHA256:7YqN4rXO08yixTeOO+UtQ8jHyIGycICuctQgRYVGnWw` | (staple the printout) |
|
||||
| S7 | ep0 read-only token `dooplex-hub@pbs!restore` | — | ______________________________________________________ |
|
||||
| S8 | Hetzner account: login e-mail · password · two-factor recovery codes | — | ______________________________________________________ |
|
||||
| S11 | Cloudflare account: login · password · two-factor recovery codes | — | ______________________________________________________ |
|
||||
| S12 | Gmail `felhom.eu@gmail.com`: password · two-factor recovery codes | — | ______________________________________________________ |
|
||||
| S19 | CC secrets age key — opens `admin/cc-secrets` (every secret Claude Code uses on DooPlex); the store is in Gitea, behind S1 | recipient `age1aw83ej37x9sck89xwt856ry0553kf5sng3w544an4et54yjdx5qq79utrv` | `AGE-SECRET-KEY-1`______________________________________ |
|
||||
| S3 | Vaultwarden master password — **in your head**; write it here only if you decide to | — | ______________________________________________________ |
|
||||
| S1 | DooPlex off-site kulcs / DooPlex off-site key (Gitea, jelszókezelő / password manager, k8s Secrets) | DooPlex `/etc/felhom-dooplex-offsite/enc.key` | `data`: ____________________________ |
|
||||
| S2 | Hub-adatbázis off-site kulcs / Hub-DB off-site key | DooPlex `/etc/felhom-hub-backup/enc.key` | `data`: ____________________________ |
|
||||
| S4 | Hub pecsétkulcs / hub seal key `OFFSITE_SECRET_KEY` | k8s Secret `felhom-system/offsite-secret-key` | __________________________________ |
|
||||
| S5 | DooPlex restic / Secrets-export jelmondat / passphrase | DooPlex `/etc/backup/restic-password` | __________________________________ |
|
||||
| S6a | Aláíró HELYREÁLLÍTÓ kulcs / signing RECOVERY key | DooPlex `/mnt/5_hdd/felhom.eu/felhom-rec-recovery` | (tűzd ide / staple) |
|
||||
| S6b | Aláíró ÜZEMI kulcs / signing OPERATIONAL key | DooPlex `/mnt/5_hdd/felhom.eu/felhom-op-operational` | (tűzd ide / staple) |
|
||||
| S7 | ep0 csak-olvasó token / read-only token `dooplex-hub@pbs!restore` | DooPlex `/etc/felhom-hub-backup/token-restore` | __________________________________ |
|
||||
| S8 | Hetzner fiók / account (e-mail · jelszó / password · 2FA kódok / codes) | Vaultwarden | __________________________________ |
|
||||
| S11 | Cloudflare fiók / account (login · jelszó / password · 2FA kódok / codes) | Vaultwarden | __________________________________ |
|
||||
| S12 | Gmail `felhom.eu@gmail.com` (jelszó / password · 2FA kódok / codes) | Vaultwarden | __________________________________ |
|
||||
| S19 | CC secrets age kulcs / key (`admin/cc-secrets`) | DooPlex `~/.config/sops/age/keys.txt`; Vaultwarden | `AGE-SECRET-KEY-1`__________________ |
|
||||
| S3 | Vaultwarden mesterjelszó / master password | a fejedben / in your head | (csak ha akarod / only if you choose) |
|
||||
|
||||
**Not secret, needed with the keys:**
|
||||
- ep0: `167.233.158.164` (Hetzner, `felhom-hetzner`); PBS datastore `felhom-offsite`, namespace `operator`;
|
||||
its certificate fingerprint `c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:60:4a:d5:56:5f:3c:fd`.
|
||||
- Restore order: `total-loss-of-dooplex.md` (in the restored Gitea, repo `felhom.eu`, `documentation/runbooks/`).
|
||||
**Print that page too** — the runbook itself is inside the copy it explains how to open.
|
||||
|
||||
**Re-print when** a key above is rotated, and once a year.
|
||||
**Nem titok, de kell / not secret, needed:** ep0 = `167.233.158.164` (Hetzner, `felhom-hetzner`), PBS `felhom-offsite`, névtér / namespace `operator`; tanúsítvány / certificate
|
||||
`c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:60:4a:d5:56:5f:3c:fd`. Kiadások másolata / release copy: ep0 `/srv/felhom-release-copy/data`.
|
||||
Sorrend / order: `total-loss-of-dooplex.md` — **nyomtasd ki azt is / print it too.** Újranyomtatás / re-print: kulcscserekor és évente / when a key rotates, and yearly.
|
||||
|
||||
@@ -19,8 +19,7 @@ build tree. **Assume all of it is gone, and the house with it.**
|
||||
| The boxes themselves | they keep running and backing up; they cannot report, update or mail until the hub is back | — |
|
||||
| The operator's workstation (if it has them) | the WireGuard peer to ep0 (`10.77.0.250`), maybe one of ep0's two root SSH keys, maybe a Bitwarden client's offline vault cache | the operator |
|
||||
|
||||
**Lost with DooPlex and NOT in any copy** (measured 2026-10-09): the container registry (rebuild the images from
|
||||
code); *(the operator signing keys were here until 2026-10-09 afternoon; they now ride the nightly copy — R-924)*; DooPlex's own SSH key (one of ep0's two root keys);
|
||||
**Lost with DooPlex and NOT in any copy** (measured 2026-10-09): *(the container registry was here until 2026-10-23: the released images and agent packages now ride a nightly copy to ep0 — D33, „Restore the release copy" below; the rest of the registry — older tags, rc builds, non-Felhom images — is still lost)*; *(the operator signing keys were here until 2026-10-09 afternoon; they now ride the nightly copy — R-924)*; DooPlex's own SSH key (one of ep0's two root keys);
|
||||
the build tree and drills (`/mnt/5_hdd/felhom.eu/{build,drills}`); `.claude-memory` (same-disk restic only); the
|
||||
local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom; Longhorn on DooPlex only).
|
||||
|
||||
@@ -66,16 +65,48 @@ local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom
|
||||
copy, use that. Every later login comes from here.
|
||||
6. **The Secrets export:** `gpg --batch --passphrase-file <S5 file> -d secrets-<stamp>.yaml.gpg > secrets.yaml` —
|
||||
every k8s Secret of the old cluster (shred it after use).
|
||||
7. **Gitea:** `gitea-restore.md` (on a normal network this time). Then rebuild the images from the code into the new
|
||||
registry: hub, controller, agent (`RUNBOOK-manual-build.md`).
|
||||
7. **Gitea:** `gitea-restore.md` (on a normal network this time). Then put the released images and agent packages
|
||||
back into its registry from ep0 — „Restore the release copy" below (same digests; no rebuild). Rebuild from code
|
||||
(`RUNBOOK-manual-build.md`) only what the copy does not hold.
|
||||
8. **The hub:** k3s, then `RUNBOOK-hub-db-offsite-backup.md` §3 with S2 and S4 — **start it with mail held**
|
||||
(§3 step 1: the `MAIL-HOLD` marker — shipped in hub 0.145.0, 2026-10-10).
|
||||
9. **DNS:** in Cloudflare (S11) point `hub.felhom.eu` (today a CNAME to `dooplex.hopto.org`) and `gitea.dooplex.hu` at
|
||||
the new place. The boxes reconnect by themselves: their API keys are in the restored hub DB.
|
||||
10. **Signing:** the keys are in the restored copy (`signing/`) and on the sheet (S6) — put them back and sign as before. Without it, no box accepts an agent update or a
|
||||
bundle until it is re-enrolled on site (`04` §4).
|
||||
11. **Backups again:** reinstall `scripts/hub-db-backup/` and `scripts/dooplex-offsite/` from the restored code, the
|
||||
tunnel and the tokens, and check the next night's copies.
|
||||
11. **Backups again:** reinstall `scripts/hub-db-backup/`, `scripts/dooplex-offsite/` and `scripts/release-copy/` from
|
||||
the restored code, the tunnel and the tokens (the release copy needs a new read-only registry token and a new SSH
|
||||
key in ep0's `felhom-relcopy` `authorized_keys`), and check the next night's copies.
|
||||
|
||||
## Restore the release copy (D33, since 2026-10-23)
|
||||
|
||||
**What it is.** Every night at 01:40 DooPlex copies the newest 3 release tags of `felhom-controller`, `felhom-hub` and
|
||||
`felhom-samba`, and the newest 5 agent versions (binary + config bundle), to ep0 `/srv/felhom-release-copy/data`
|
||||
(`scripts/release-copy/`). Images are in skopeo's `dir:` layout — the registry's own bytes — so a push back gives the
|
||||
**same digest** as the release. Nothing in it is secret; it is not encrypted. Alarm: `FelhomReleaseCopyStale`.
|
||||
|
||||
1. **Get it off ep0** (ep0 root — step 2): `rsync -a root@167.233.158.164:/srv/felhom-release-copy/data/ ./release-copy/`
|
||||
(or work on ep0 itself). Then `cd release-copy && sha256sum -c --quiet SHA256SUMS` — every line must pass.
|
||||
2. **Images back into the new registry**, one per tag (log in to the new Gitea first: `skopeo login <gitea host>`):
|
||||
```bash
|
||||
for d in images/*/*; do repo=${d#images/}; repo=${repo%%/*}; tag=${d##*/}
|
||||
skopeo copy --all --preserve-digests "dir:$d" "docker://<gitea host>/admin/$repo:$tag"
|
||||
echo "$repo:$tag $(skopeo inspect --raw "docker://<gitea host>/admin/$repo:$tag" | sha256sum | cut -c1-64) want $(cut -c8- "$d/DIGEST")"
|
||||
done
|
||||
```
|
||||
The two hashes on each line must be equal.
|
||||
3. **Agent packages back** (Gitea's generic registry, a token with package write):
|
||||
```bash
|
||||
for d in agent/*; do v=${d#agent/}; for f in felhom-agent felhom-config-bundle.json; do
|
||||
curl -fsS -u admin:"$(cat token-file)" -X PUT --upload-file "$d/$f" \
|
||||
"https://<gitea host>/api/packages/admin/generic/felhom-agent/$v/$f"; done; done
|
||||
```
|
||||
Check each download's sha256 against `agent/<v>/<file>.sha256`.
|
||||
4. **Keep the registry's name** (`gitea.dooplex.hu`, step 9's DNS) — the boxes pull `gitea.dooplex.hu/admin/…` by name.
|
||||
|
||||
**Proven 2026-10-23** (`audits/day-2026-10-23/08-release-copy-D33.txt`): the copy was pulled back from ep0, every
|
||||
sha256 matched, every image re-verified against its digest, and controller 0.340.0 from it loaded on a scratch box
|
||||
(image ID = its config digest; a container ran). **Not yet walked:** step 2 into a real new registry, and step 3.
|
||||
|
||||
## Walked on paper (2026-10-09): what still needs DooPlex if the operator has only the sheet, memory and a new machine
|
||||
|
||||
@@ -90,7 +121,7 @@ local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom
|
||||
| the hub's seal key | S4 (or S1+S5) | sheet, or via S5 |
|
||||
| the hub DB | S2 (or Vaultwarden) | sheet, or via S3 |
|
||||
| sign a box update | S6 | **sheet — today exists on DooPlex only** |
|
||||
| hub image | the registry (lost) | rebuild from restored code — needs a Go/Docker build machine, no secret |
|
||||
| hub, controller, samba images; agent packages | the release copy on ep0 (D33, nightly since 2026-10-23) | ep0 root (S8/S9) → „Restore the release copy" — no secret, no rebuild |
|
||||
| Gitea `main` → `felhom.eu` website, ISO host | Cloudflare (S11), the build tree (lost) | S11 on the sheet; the build tree is rebuilt from code |
|
||||
| Authentik (S17) | not in any copy | not needed for the business; DooPlex homelab apps that use it are out of scope |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user