night 2026-10-23 so far: D30 live proof (re-key → new-key handshake; ep0-started handshakes heal a short clock jump by themselves), D4 patches v2 after review, test-night install/fixture evidence, DooPlex survival survey (read only) + PLAN, 03/05/07 + CONTEXT, total-loss runbook: MAIL-HOLD shipped
gates / gates (push) Successful in 15m49s
gates / gates (push) Successful in 15m49s
This commit is contained in:
+19
@@ -16,6 +16,25 @@
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **2026-10-22 evening — operator rulings D29, D30, D31 (CC's picks accepted; the decide rows now in `STATUS-HISTORY.md`).**
|
||||
> **D29** = no: the controller repository stays public for now (R-925). **D30** = the agent makes a new WireGuard key and
|
||||
> registers it on the existing peer path (R-1204). **D31** = the hub skips a deleted install that never made a key when it
|
||||
> decides the automatic PBS re-issue (R-1205). If the operator reverses one in the morning, its commit is reverted before
|
||||
> release (night brief 2026-10-23 §0).
|
||||
|
||||
> **2026-10-22/23 night — what was built (night brief; nothing released, deployed, published or vouched).**
|
||||
> **Stray-copy fix** (R-1214/R-1180): controller `66a69d4` + `912e592` — the capped copy's container has a fixed name +
|
||||
> label `felhom.tier2-copy=1`, is removed by name on any error, every leftover is removed at start before the scheduler
|
||||
> and web server, a restore is refused while a copy runs (`backup.Tier2CopyBlocksRestore`, in-memory record + a stray
|
||||
> flag until the start-up removal succeeds); `--mount`, image ID, `--pull never`, `--no-healthcheck`. Proven on 9202 rc.5.
|
||||
> **D30:** agent `ae69ad4` (`wgtunnel/rekey.go`, 15 min, once an hour, `private.key.prev`); hub `84751cb0` pins the
|
||||
> alarm clearing. **D31:** hub `84751cb0` `store.LatestKeyHoldingHostDeletion` (hub 0.167.0 source, NOT built — a hub
|
||||
> image build is the morning's, with the deploy). **Review fixes:** controller `943c517` (C5 catch-up releases the
|
||||
> whole-guest hold before its checks; C3 text; C6/C8 wording); the D4 patches v2 (`audits/night-2026-10-23/P3-review/`).
|
||||
> **Images:** controller `0.340.0-rc.5` (= `66a69d4`) and `0.340.0-rc.6` (= `943c517`); no hub image tonight.
|
||||
> **Survey:** `audits/dooplex-survival-2026-10-22/PLAN.md` (G1–G10; the three biggest: the sheet, the registry, the
|
||||
> home line). Report: `REPORT-night-2026-10-23.md`.
|
||||
|
||||
> **2026-10-22 15:47 — operator rulings, „all picks" (the decide table, rows now in `STATUS-HISTORY.md`).**
|
||||
> **D3** = change the zero-knowledge wording now, look again after the first sale (R-985). **D4** = root SSH from the
|
||||
> LAN by key only; the console password stays (R-982). **D5** = approve the minimum machine, `audits/sizing-2026-10-13/SIZING.md`
|
||||
|
||||
@@ -131,7 +131,20 @@ the one trust root; the agent's API token is untouched; a box gets the new rule
|
||||
|
||||
**[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — built as a ready patch, NOT shipped] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag.
|
||||
|
||||
**[FACT, agent main `a0c88e5`, unreleased] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b). The repair is STATUS decision D30.
|
||||
**[FACT, agent main `a0c88e5`, unreleased] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b).
|
||||
|
||||
**[DESIGN, operator ruling D30 2026-10-22; agent main `ae69ad4`, unreleased] The tunnel heals itself by a new key.** When
|
||||
the latest handshake stays stamped in the future for 15 minutes, the agent makes a new WireGuard key pair (the old
|
||||
private key kept as `wg/private.key.prev`, 0600 — the escrow may hold it) and registers the new public key on the
|
||||
existing path, `POST /hosts/<id>/wg`: the hub re-keys the peer in place (same /32, generation bump) and its full-list
|
||||
push to ep0 (`wgsync`, `wg syncconf`) drops the old key and adds the new one. A new key is a new peer for ep0, with no
|
||||
stored timestamp. Once an hour at most; not while the unit is down; never during a re-register already in flight. The
|
||||
interface switches at the agent's next desired-state fetch (up to its report interval). **`[FACT]` 2026-10-23 night
|
||||
(test box Tester4-16fdef, `audits/night-2026-10-23/P2-D30/`):** with ep0 free to start a handshake of its own, a 2-hour
|
||||
clock jump healed by itself in ~3 minutes (ep0 had traffic for the box; the box accepts ep0's timestamps) and the agent
|
||||
correctly did not re-key; with ep0's handshake starts dropped on the box (the rehearsal's condition), the tunnel stayed
|
||||
dead and the agent re-keyed after 15 minutes. **Known gap:** a reboot after the clock was corrected forgets the
|
||||
future stamp (R-1223).
|
||||
|
||||
## 4. Control model — reconcile + signed destructive ops
|
||||
|
||||
|
||||
@@ -371,6 +371,7 @@ token id, fingerprint, datastore, tunnel IP, `secret_generation`) with its consu
|
||||
| **re-issue, descriptor ABSENT, DR flag on** (R-511, v0.114.0) | **adopted**: re-keyed | **rebuilt** from the endpoint's answer; `pbsdr_adopted` audit row |
|
||||
| host delete | **kept** (tenancy survives) | goes with the host |
|
||||
| host delete acknowledged through escrow-ack, then a new box | re-keyed automatically (F-14) | rebuilt |
|
||||
| …and a later install deleted before it made a key, then a new box (D31, operator 2026-10-22; hub 0.167.0 source) | re-keyed automatically — the gate reads the newest deletion of a host that HELD A KEY (`LatestKeyHoldingHostDeletion`) | rebuilt |
|
||||
| RESET / customer delete | **deprovisioned** — namespace, every backup group AND token | purged |
|
||||
|
||||
**Why adopt exists:** a rebuilt box's WG hook refused („the endpoint already holds a PBS token … use
|
||||
@@ -459,6 +460,12 @@ date taken) are shown on its own app page, on its box, until it dismisses them
|
||||
|
||||
**[FACT, hub `e267c5e2`]** The host report carries the agent's `wireguard` block (`hostWireguard`). `checkWGTunnel` (`api/host_tunnel.go`, called after `checkHostSwap`) calls the tunnel stale when the box reports `handshake_future_s` > 0 (the clock-ahead lock-out) or a last handshake older than 6 h (healthy boxes measured at most 113–124 s over 24 h). It is edge-triggered against the host's latest tunnel event (`store.LatestHostTunnelEvent`): `wg_tunnel_stale` once on the way down, `wg_tunnel_ok` once on the way back; both are allow-listed and operator-only (`notify.operatorOnlyEvents`, `wg_tunnel_operator_only_test.go`) — the household cannot repair a tunnel. A box with no `wireguard` block (an agent before the field, or no tunnel) raises nothing.
|
||||
|
||||
**[FACT, hub `84751cb0`, 0.167.0 source] The repair clears it (D30).** An agent with D30 (`03` §R-1204) registers a new key
|
||||
after 15 minutes of a future-stamped handshake; the new key's first handshake makes the report fresh, and the alarm
|
||||
records `wg_tunnel_ok` once (`TestD30_TheAlarmClearsWhenTheNewKeyShakesHands`). `wg_tunnel_ok` is an info event:
|
||||
stored, not mailed. **Known gaps (rows):** a reboot after the clock was corrected sends no handshake at all and is not
|
||||
judged (R-1223); the operator's mail cooldown is per customer, not per host (R-1224).
|
||||
|
||||
### 15.5 How "the Hungarian did not change" is known
|
||||
|
||||
56 goldens captured from v0.117.0 before any string moved, in
|
||||
|
||||
@@ -1590,6 +1590,21 @@ or pattern. **Why:** the data-loss hunt of 2026-10-20 proved two nights that del
|
||||
(R-1175: one share group's reconcile removed the other groups' copies, and a dead drive's copy; R-1181: a missing or
|
||||
emptied folder took its copy within one night). Controller: `internal/backup/tier2_kept.go`.
|
||||
|
||||
### 8.2b The second-drive copy runs in a memory-capped container that never outlives the copy [DESIGN + FACT, R-1214, 2026-10-22/23 nights; rides controller 0.340.0]
|
||||
|
||||
The copy (`rsync -a --delete`) runs in a container of the controller's own image, capped at 512 MB, so its page cache
|
||||
is reclaimed inside the cap instead of from the apps (R-1214: full stall 5.7 s → 0.37 s on 9202). Only for paths the
|
||||
controller and the Docker daemon see alike (`/mnt/…`, no symlink); the daemon must see the path itself (`--mount`
|
||||
refuses a missing one — then the in-process copy runs). **The container must never outlive the copy** (the stray-copy
|
||||
fix, 2026-10-23 night): a fixed name and the label `felhom.tier2-copy=1`; on any error, the timeout included, it is
|
||||
removed by name; at controller start, before the scheduler and the web server, every labelled container is removed;
|
||||
and a restore is refused while any second-drive copy runs or a leftover may still run (the other direction of R-1180 —
|
||||
a `--delete` beside a restore removes what the restore brings back). **`[FACT]` 9202, rc.5
|
||||
(`audits/night-2026-10-23/P1-straycopy/`):** a controller restart in the middle of a 16 GB copy → the new controller
|
||||
removed the old container at start; a restore probe during the copy was refused; the redone copy equals the source; a
|
||||
15-second test limit → the container removed, the partial copy stopped growing. Controller:
|
||||
`internal/backup/tier2_copy_lifecycle.go`.
|
||||
|
||||
### 8.1 The blank cells, listed explicitly
|
||||
|
||||
Per the rule that a blank is a finding, here they are:
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
# DooPlex survival survey — night 2026-10-22 → 23 (read only)
|
||||
|
||||
Night brief Part 5. **Nothing on DooPlex was changed.** Every number below was read on DooPlex tonight (unit state,
|
||||
journal lines, `du`, `kubectl get`, DNS, HTTP status) — raw readings in `readings/`. Secrets are named, never read:
|
||||
no key file, token file or password was opened. ep0 was not contacted (the nightly jobs' own journal lines are the
|
||||
evidence of what reached it). Read with `runbooks/total-loss-of-dooplex.md` (the order of steps) and the 2026-10-09
|
||||
plan (`audits/dooplex-survival-2026-10-09/PLAN.md`, which built the nightly copy).
|
||||
|
||||
## 1. What lives on DooPlex, and what saves it
|
||||
|
||||
| Thing | Where it lives | What backs it up | Where the copy is | Last copy (measured) | Who can restore | How long |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **Gitea** — 12 repositories (845 MB), DB in CNPG, `app.ini` | PVC `gitea-data` (Longhorn, `sdb1`) | (1) 6-hourly `pg_dump` + Longhorn backup, both on `sda1` (same box); (2) `felhom-dooplex-offsite` nightly 00:20 | ep0, namespace `operator`, `host/dooplex-gitea`, encrypted (key **S1**) | 2026-10-22 00:22, 736 MiB (425 MiB new), 42 s, success | the operator with S1 + the read token S7 (or ep0 root) | restore ~1 min + start ~1 min on a ready machine (walked 2026-10-09 Part C); on new hardware add k3s/Docker install — hours |
|
||||
| **Weekly proof of that copy** | DooPlex timer, Sunday 05:30 | — | — | 2026-10-18: 27 739 files match, **11** repositories pass `git fsck`, `gitea.dump` readable, Vaultwarden 1 user / 810 items | — | — |
|
||||
| **Container registry + generic packages** (controller/hub images, agent packages) | `/data/gitea/packages` in `gitea-data`, **38.6 GB** | Longhorn backup on `sda1` only | **nowhere off DooPlex** (left out on purpose 2026-10-09: „rebuilt from the code") | — | rebuild from the restored code on a build machine | never timed |
|
||||
| **Hub + its DB** (`hub.db` ~493 MiB) | PVC `hub-data` (Longhorn) | Longhorn backup (same box); `felhom-hub-db-backup` nightly 02:30 | ep0 `operator`, `host/dooplex-hub`, encrypted (key **S2**); seal key **S4** needed to open its sealed passwords | 2026-10-22 02:31, 493 MiB (24 MiB compressed), 6 s, success | the operator with S2 + S4 + S7 | ~5 min on a bench with k3s ready (2026-10-09 Part D) |
|
||||
| **Weekly proof of that copy** | Sunday 04:31 | — | — | 2026-10-18: integrity ok, 4 hosts, 4 sealed console passwords, 0 readable | — | — |
|
||||
| **Operator signing keys** (3 files, 0600, no passphrase) | `/mnt/5_hdd/felhom.eu/` on `sda1` (the same disk as the local backup tree) | inside the nightly `host/dooplex-gitea` copy (`signing/`; a missing key refuses the push) | ep0, behind S1; paper (S6) **if printed** | with the Gitea copy (2026-10-22) | the operator with S1, or the printout | minutes |
|
||||
| **Password manager** (Vaultwarden, 1 user, 810 items) | PVC `vaultwarden-data` (Longhorn) | Longhorn backup (same box); nightly copy (its own `vaultwarden backup` + `rsa_key.pem`) | ep0, behind S1; the vault itself opens with the master password S3 | with the Gitea copy (2026-10-22) | the operator with S1 + S3 | started once on a throwaway (2026-10-09) |
|
||||
| **k8s Secrets** (every Secret, GPG-encrypted with S5) | nightly export `/mnt/5_hdd/backup/secrets` (136 MB, 30 days) | the same file rides the nightly copy | ep0 behind S1, then S5 | 2026-10-22 | the operator with S1 + S5 | minutes |
|
||||
| **CI** (`act-runner` pod, PVC `act-runner-data`) | k3s | nothing needed: re-registered from Gitea | — | — | the operator | ~15 min |
|
||||
| **Website, installer script, `hub.felhom.eu`, `gitea.dooplex.hu`** | DooPlex, behind the home line (all four resolve to the home address; measured) | the website and installer are code in the Gitea copy | ep0 (as code) | — | the operator + a DNS change at Cloudflare (S11) | after Gitea + k3s |
|
||||
| **The ISO** | Cloudflare R2 (`iso.felhom.eu`) | — | not on DooPlex | — | — | survives |
|
||||
| **ArgoCD** | k3s; Applications from `homelab-manifests` (in the Gitea copy) | its Secrets in the export | ep0 behind S1 + S5 | — | the operator | ~30 min; **the per-app auto-sync settings are set on the live objects, not in git** (several apps differ) |
|
||||
| **DooPlex's own local backups** (R-232) | `/mnt/5_hdd/backup` on `sda1`: Longhorn 117 GB, `data` 53 GB, PostgreSQL dumps 2.1 GB, `ep0-copy` 21 GB, … | `dooplex-backup` nightly 03:00 | **the same box** | 2026-10-22 03:16, „All backup components completed successfully" | — | — |
|
||||
|
||||
**Restore tests:** both weekly tests passed on 2026-10-18 (their next run is Sunday 2026-10-25). Since then the pod
|
||||
lists **12** repositories against the test's 11; the nightly push refuses a copy with fewer repositories than the pod
|
||||
lists, and it succeeded, so the twelfth is in the copy — the next Sunday test will show it.
|
||||
|
||||
## 2. `total-loss-of-dooplex.md`, step by step — works today / stale / never tried
|
||||
|
||||
| Step | Verdict | Why |
|
||||
|---|---|---|
|
||||
| 1. A machine and the sheet | **depends** | the sheet's printing is the operator's (R-923, R-924 „LEFT"); CC cannot see whether it exists |
|
||||
| 2. Reach ep0's PBS | works **from DooPlex** (the nightly jobs use the tunnel); **never tried** from new hardware (WireGuard peer, Hetzner console) | |
|
||||
| 3. A read token (S7) | works today | the weekly restore tests use it |
|
||||
| 4. Restore `host/dooplex-gitea` with S1 | works today | the weekly test restores and checks it (2026-10-18) |
|
||||
| 5. Vaultwarden first | works (once) | started on a throwaway 2026-10-09; its rows are checked weekly |
|
||||
| 6. The Secrets export with S5 | works (once) | opened 2026-10-09; **the form of S5's offline copy is unknown to CC** |
|
||||
| 7. Gitea, then rebuild the images | Gitea: works (once, 2026-10-09). Image rebuild: **never tried** | the registry is not copied |
|
||||
| 8. The hub with S2 + S4, mail held | works (once, 2026-10-09 Part D) | **stale text:** it says the mail hold comes „from the next hub release" — it shipped in hub 0.145.0 (2026-10-10); fixed tonight |
|
||||
| 9. DNS at Cloudflare | **never tried** | |
|
||||
| 10. Signing | keys readable (the weekly test derives each public half) — **signing from the restored keys never tried** | |
|
||||
| 11. Backups again | **never tried** on new hardware | `install.sh` exists in the code |
|
||||
|
||||
## 3. The gaps — two options each, with CC's pick
|
||||
|
||||
| # | Gap | Option A (cost: money / operator hours / risk) | Option B (cost) | CC's pick |
|
||||
|---|---|---|---|---|
|
||||
| G1 | **The paper sheet** — S1 opens every off-site copy (Gitea, Vaultwarden, signing keys, Secrets); without it no copy opens (R-923, R-924) | print `break-glass-sheet.md` filled (S1, S2, S4, S5, S6, S7, S8, S11, ep0 fingerprint) and store it outside the home, sealed — 0 € / ~30 min / paper can be lost or read | the key files on an encrypted USB stick kept outside the home + its passphrase in the head — ~10 € / ~1 h / one more secret to remember, a stick can die | **A** — it is the written plan; nothing else breaks the circle |
|
||||
| G2 | **Signing keys without a passphrase** (R-924) | keep as the custody ruling (2026-09-16) says until the first sale; they ride the S1-encrypted copy — 0 € / 0 h / a DooPlex break-in can sign | a passphrase on the operational key and the recovery key off DooPlex — 0 € / ~2 h / every signed job then needs the operator; no unattended signing | **A** now; revisit at the first sale (the ruling stands) |
|
||||
| G3 | **The password manager holds keys to DooPlex's own copies** (R-923) | the sheet (G1) breaks the circle — no extra cost | a monthly encrypted export to the operator's laptop — 0 € / 10 min a month / easy to forget | **A** (same act as G1) |
|
||||
| G4 | **DooPlex's backups sit inside the same box** (R-232) — the Felhom parts are off-site now; the homelab apps, Authentik, Prometheus are not | accept for the homelab (not Felhom) — 0 € / 0 h / a homelab loss is total | restic of the chosen `/mnt/5_hdd/backup` sets to a Storage Box sub-account — ~4 €/month / ~2 h / ~170 GB first upload | **A** for Felhom's scope; the homelab is the operator's call |
|
||||
| G5 | **Hub DB copy is up to 24 h old** — a box that enrolled or re-escrowed after 02:30 has no escrow row in a restored hub | keep nightly — 0 € / 0 h / up to a day of hub changes lost | push every 6 h (same job, same token, same prune) — 0 € / ~30 min (a DooPlex change, operator's yes) / ~4 × 24 MB a day on ep0 | **B** |
|
||||
| G6 | **Gitea repositories off-site** — covered nightly, tested weekly | keep — 0 € | a second off-site mirror (Codeberg/GitHub private) — 0 € / ~1 h / code at a third party | **A** |
|
||||
| G7 | **The registry and the agent packages exist only on DooPlex** — every box update, every reinstall and every new install pull from them | copy only what a box can be offered (the released controller/hub tags, the agent packages, the infra images) nightly to ep0 `operator` — 0 € / ~2 h CC + the operator's yes for a DooPlex change / a few GB on ep0 | a timed rebuild-from-code drill, then accept the outage — 0 € / ~3 h once / a rebuilt image is not the same digest (the „a version is pushed once" rule) | **A** |
|
||||
| G8 | **Website, installer script and the hub all on the home line** — a DooPlex loss stops new installs and the public site | accept: Gitea and the hub come back first, then the site — 0 € / hours on the day / no installs meanwhile | serve the static site and the installer tag from Cloudflare (as the ISO already is) — 0 € / ~2 h / a second publish path to keep true | **B**, after the release week |
|
||||
| G9 | **ArgoCD's per-app sync settings live only on the cluster** | write them into `homelab-manifests` — 0 € / ~30 min / none | re-set by hand after a loss from the memory notes — 0 € / ~1 h on the day | **A** |
|
||||
| G10 | **S5's offline copy: form unknown to CC** | put S5 on the sheet (G1) — 0 € | re-encrypt the Secrets export with an S1-derived key — 0 € / ~1 h / a DooPlex change | **A** |
|
||||
|
||||
**The three biggest:** G1 (no sheet → nothing opens), G7 (no registry copy → no update or install after a loss),
|
||||
G8 (the website, installer and hub on one home line → new installs stop for hours or days).
|
||||
|
||||
## 4. What tomorrow's hour decides (also in `STATUS.md`)
|
||||
|
||||
1. G1 — print the sheet now? (CC: yes)
|
||||
2. G7 — copy the release images and agent packages to ep0 nightly? (CC: yes)
|
||||
3. G5 — hub DB every 6 hours? (CC: yes)
|
||||
4. G8 — the site and installer on Cloudflare after the release week? (CC: yes, later)
|
||||
@@ -0,0 +1,7 @@
|
||||
=== felhom-dooplex-offsite: success Thu 2026-10-22 00:22:53 CEST 0
|
||||
=== felhom-hub-db-backup: success Thu 2026-10-22 02:31:31 CEST 0
|
||||
=== dooplex-backup: success Thu 2026-10-22 03:16:26 CEST 0
|
||||
=== felhom-hub-db-restore-test: success Sun 2026-10-18 04:31:19 CEST 0
|
||||
=== felhom-dooplex-offsite-restore-test: success Sun 2026-10-18 05:33:45 CEST 0
|
||||
=== backup-freshness: success Thu 2026-10-22 21:30:01 CEST 0
|
||||
=== felhom-ep0-copy-gc: success Thu 2026-10-22 08:00:00 CEST 0
|
||||
@@ -0,0 +1,43 @@
|
||||
=== felhom-dooplex-offsite
|
||||
2026-10-22T00:22:09+02:00 dooplex felhom-dooplex-offsite[1390185]: Downloading previous manifest (Wed Oct 21 00:21:51 2026)
|
||||
2026-10-22T00:22:09+02:00 dooplex felhom-dooplex-offsite[1390185]: Upload directory '/var/lib/felhom-dooplex-offsite/stage/root' to 'dooplex-hub@pbs!push@127.0.0.1:18007:felhom-offsite' as dooplex.pxar.didx
|
||||
2026-10-22T00:22:49+02:00 dooplex felhom-dooplex-offsite[1390185]: dooplex.pxar: had to backup 424.942 MiB of 736.086 MiB (compressed 415.906 MiB) in 40.05 s (average 10.611 MiB/s)
|
||||
2026-10-22T00:22:49+02:00 dooplex felhom-dooplex-offsite[1390185]: dooplex.pxar: backup was done incrementally, reused 311.144 MiB (42.3%)
|
||||
2026-10-22T00:22:49+02:00 dooplex felhom-dooplex-offsite[1390185]: Uploaded backup catalog (1.402 MiB)
|
||||
2026-10-22T00:22:51+02:00 dooplex felhom-dooplex-offsite[1390185]: Duration: 42.46s
|
||||
2026-10-22T00:22:51+02:00 dooplex felhom-dooplex-offsite[1390185]: End Time: Thu Oct 22 00:22:51 2026
|
||||
2026-10-22T00:22:51+02:00 dooplex felhom-dooplex-offsite[1375323]: felhom-dooplex-offsite: pushed to ep0 (ns operator, host/dooplex-gitea) in 42 s
|
||||
2026-10-22T00:22:51+02:00 dooplex felhom-dooplex-offsite[1375323]: felhom-dooplex-offsite: success signal written
|
||||
2026-10-22T00:22:53+02:00 dooplex systemd[1]: felhom-dooplex-offsite.service: Deactivated successfully.
|
||||
2026-10-22T00:22:53+02:00 dooplex systemd[1]: Finished felhom-dooplex-offsite.service - Felhom: push Gitea + DooPlex secrets to ep0, encrypted (R-232).
|
||||
2026-10-22T00:22:53+02:00 dooplex systemd[1]: felhom-dooplex-offsite.service: Consumed 36.202s CPU time, 1.6G memory peak.
|
||||
=== felhom-hub-db-backup
|
||||
2026-10-22T02:31:18+02:00 dooplex felhom-hub-db-backup[2573155]: Encryption key fingerprint: b2:19:bf:36:3b:97:3d:6c
|
||||
2026-10-22T02:31:18+02:00 dooplex felhom-hub-db-backup[2573155]: Downloading previous manifest (Wed Oct 21 02:30:55 2026)
|
||||
2026-10-22T02:31:18+02:00 dooplex felhom-hub-db-backup[2573155]: Upload directory '/var/lib/felhom-hub-backup/stage' to 'dooplex-hub@pbs!push@127.0.0.1:18007:felhom-offsite' as hubdb.pxar.didx
|
||||
2026-10-22T02:31:24+02:00 dooplex felhom-hub-db-backup[2573155]: hubdb.pxar: had to backup 492.879 MiB of 492.879 MiB (compressed 23.873 MiB) in 5.17 s (average 95.306 MiB/s)
|
||||
2026-10-22T02:31:24+02:00 dooplex felhom-hub-db-backup[2573155]: Uploaded backup catalog (56 B)
|
||||
2026-10-22T02:31:24+02:00 dooplex felhom-hub-db-backup[2573155]: Duration: 5.76s
|
||||
2026-10-22T02:31:24+02:00 dooplex felhom-hub-db-backup[2573155]: End Time: Thu Oct 22 02:31:24 2026
|
||||
2026-10-22T02:31:24+02:00 dooplex felhom-hub-db-backup[2570694]: felhom-hub-db-backup: pushed hub-20261022T000000Z.db to ep0 (ns operator) in 6 s
|
||||
2026-10-22T02:31:24+02:00 dooplex felhom-hub-db-backup[2570694]: felhom-hub-db-backup: success signal written
|
||||
2026-10-22T02:31:31+02:00 dooplex systemd[1]: felhom-hub-db-backup.service: Deactivated successfully.
|
||||
2026-10-22T02:31:31+02:00 dooplex systemd[1]: Finished felhom-hub-db-backup.service - Felhom: push the hub DB snapshot to ep0 (R-173).
|
||||
2026-10-22T02:31:31+02:00 dooplex systemd[1]: felhom-hub-db-backup.service: Consumed 16.683s CPU time, 766M memory peak.
|
||||
=== dooplex-backup
|
||||
2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2932973]: 115G /mnt/5_hdd/backup/longhorn-pvc
|
||||
2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2932973]: 15G /mnt/5_hdd/backup/minecraft-tlauncher
|
||||
2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2932973]: 35M /mnt/5_hdd/backup/paperless
|
||||
2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2932973]: 2.1G /mnt/5_hdd/backup/postgresql
|
||||
2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2932973]: 992K /mnt/5_hdd/backup/romm
|
||||
2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2932973]: 136M /mnt/5_hdd/backup/secrets
|
||||
2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2932973]: 4.8M /mnt/5_hdd/backup/sparkyfitness
|
||||
2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2970506]: [2026-10-22 03:16:26] [INFO] All backup components completed successfully!
|
||||
2026-10-22T03:16:26+02:00 dooplex backup-all.sh[2970509]: [2026-10-22 03:16:26] [INFO] ========================================================
|
||||
2026-10-22T03:16:26+02:00 dooplex systemd[1]: dooplex-backup.service: Deactivated successfully.
|
||||
2026-10-22T03:16:26+02:00 dooplex systemd[1]: Finished dooplex-backup.service - Dooplex Cluster Backup.
|
||||
2026-10-22T03:16:26+02:00 dooplex systemd[1]: dooplex-backup.service: Consumed 5min 2.211s CPU time, 1.9G memory peak.
|
||||
=== felhom-hub-db-restore-test
|
||||
-- No entries --
|
||||
=== felhom-dooplex-offsite-restore-test
|
||||
-- No entries --
|
||||
@@ -0,0 +1,31 @@
|
||||
=== felhom-hub-db-restore-test (last run)
|
||||
2026-10-18T04:31:08+02:00 dooplex felhom-hub-db-restore-test[3899842]: Using encryption key from '/etc/felhom-hub-backup/enc.key'..
|
||||
2026-10-18T04:31:08+02:00 dooplex felhom-hub-db-restore-test[3899842]: Fingerprint: b2:19:bf:36:3b:97:3d:6c
|
||||
2026-10-18T04:31:13+02:00 dooplex felhom-hub-db-restore-test[3899842]: restore complete (458.028 MiB processed in 4.9s, average 93.178 MiB/s)
|
||||
2026-10-18T04:31:13+02:00 dooplex felhom-hub-db-restore-test[3899712]: felhom-hub-db-restore-test: checked: integrity ok, 4 host(s), 4 sealed console password(s), 0 readable
|
||||
2026-10-18T04:31:13+02:00 dooplex felhom-hub-db-restore-test[3899712]: felhom-hub-db-restore-test: success signal written
|
||||
2026-10-18T04:31:19+02:00 dooplex systemd[1]: felhom-hub-db-restore-test.service: Deactivated successfully.
|
||||
2026-10-18T04:31:19+02:00 dooplex systemd[1]: Finished felhom-hub-db-restore-test.service - Felhom: restore-test the hub DB copy on ep0 (R-173).
|
||||
2026-10-18T04:31:19+02:00 dooplex systemd[1]: felhom-hub-db-restore-test.service: Consumed 5.907s CPU time, 821.2M memory peak.
|
||||
=== felhom-dooplex-offsite-restore-test (last run)
|
||||
2026-10-18T05:32:13+02:00 dooplex felhom-dooplex-offsite-restore-test[269573]: progress 46% (291.015 MiB of 625.417 MiB in 36.5s, 5.911 MiB/s)
|
||||
2026-10-18T05:32:53+02:00 dooplex felhom-dooplex-offsite-restore-test[269573]: progress 77% (485.113 MiB of 625.417 MiB in 1m 17.3s, 4.763 MiB/s)
|
||||
2026-10-18T05:33:21+02:00 dooplex felhom-dooplex-offsite-restore-test[269573]: restore complete (625.417 MiB processed in 1m 45.4s, average 5.935 MiB/s)
|
||||
2026-10-18T05:33:44+02:00 dooplex felhom-dooplex-offsite-restore-test[269413]: felhom-dooplex-offsite-restore-test: checked: 27739 files match the manifest, 11 repositories pass git fsck, gitea.dump readable, 3 secrets file(s), Vaultwarden 1 user(s) / 810 item
|
||||
2026-10-18T05:33:44+02:00 dooplex felhom-dooplex-offsite-restore-test[269413]: felhom-dooplex-offsite-restore-test: success signal written
|
||||
2026-10-18T05:33:45+02:00 dooplex systemd[1]: felhom-dooplex-offsite-restore-test.service: Deactivated successfully.
|
||||
2026-10-18T05:33:45+02:00 dooplex systemd[1]: Finished felhom-dooplex-offsite-restore-test.service - Felhom: restore-test the Gitea + secrets copy on ep0 (R-232).
|
||||
2026-10-18T05:33:45+02:00 dooplex systemd[1]: felhom-dooplex-offsite-restore-test.service: Consumed 35.494s CPU time, 931.3M memory peak.
|
||||
=== offsite stage (names + sizes, no contents)
|
||||
du: cannot access '/var/lib/felhom-dooplex-offsite/stage/root/*': No such file or directory
|
||||
ls: cannot access '/var/lib/felhom-dooplex-offsite/stage/root/signing': No such file or directory
|
||||
felhom-backup-failmail
|
||||
felhom-backup-failmail@.service
|
||||
felhom-dooplex-offsite
|
||||
felhom-dooplex-offsite-restore-test
|
||||
felhom-dooplex-offsite-restore-test.service
|
||||
felhom-dooplex-offsite-restore-test.timer
|
||||
felhom-dooplex-offsite.service
|
||||
felhom-dooplex-offsite.timer
|
||||
install.sh
|
||||
test_dooplex_offsite.py
|
||||
@@ -0,0 +1,11 @@
|
||||
=== felhom-ep0-copy-gc last run
|
||||
2026-10-22T08:00:00+02:00 dooplex felhom-ep0-copy-gc[1347525]: ep0-copy-gc: done: 6 copy namespace(s), 6 on ep0, 0 absent, 0 to delete, 0 failed
|
||||
2026-10-22T08:00:00+02:00 dooplex systemd[1]: Finished felhom-ep0-copy-gc.service - Felhom: remove a deleted customer's namespace from ep0-copy (R-901, decision 181).
|
||||
=== gitea repos (pod listing)
|
||||
pod gitea-system/gitea-5dbcbf7c7b-4j8qp
|
||||
12
|
||||
845.4M /data/git/repositories
|
||||
38.6G /data/gitea/packages
|
||||
=== backup-freshness unit
|
||||
Description=Backup freshness textfile metrics for localhost
|
||||
ExecStart=/usr/local/sbin/backup-freshness.sh
|
||||
@@ -0,0 +1,32 @@
|
||||
=== /mnt/5_hdd/backup (names, sizes)
|
||||
53G /mnt/5_hdd/backup/data
|
||||
25G /mnt/5_hdd/backup/demo-felhom
|
||||
21G /mnt/5_hdd/backup/ep0-copy
|
||||
69M /mnt/5_hdd/backup/homelab-manifests
|
||||
41M /mnt/5_hdd/backup/jarrs.eu
|
||||
178M /mnt/5_hdd/backup/k3s
|
||||
2.5M /mnt/5_hdd/backup/logs
|
||||
117G /mnt/5_hdd/backup/longhorn-pvc
|
||||
15G /mnt/5_hdd/backup/minecraft-tlauncher
|
||||
35M /mnt/5_hdd/backup/paperless
|
||||
2.1G /mnt/5_hdd/backup/postgresql
|
||||
992K /mnt/5_hdd/backup/romm
|
||||
136M /mnt/5_hdd/backup/secrets
|
||||
4.8M /mnt/5_hdd/backup/sparkyfitness
|
||||
=== longhorn-pvc sets (names)
|
||||
backupstore
|
||||
=== signing key files (names, modes, no contents)
|
||||
-rw------- kisfenyo /mnt/5_hdd/felhom.eu/felhom-op-operational
|
||||
-rw-rw-r-- kisfenyo /mnt/5_hdd/felhom.eu/felhom-op-operational.pub
|
||||
-rw------- kisfenyo /mnt/5_hdd/felhom.eu/felhom-rec-recovery
|
||||
-rw-rw-r-- kisfenyo /mnt/5_hdd/felhom.eu/felhom-rec-recovery.pub
|
||||
-rw------- kisfenyo /mnt/5_hdd/felhom.eu/felhom_op_ed25519
|
||||
-rw-rw-r-- kisfenyo /mnt/5_hdd/felhom.eu/felhom_op_ed25519.pub
|
||||
=== same disk? backup tree vs sources
|
||||
Filesystem Mounted on
|
||||
/dev/sda1 /mnt/5_hdd
|
||||
/dev/sda1 /mnt/5_hdd
|
||||
/dev/sdb1 /
|
||||
/dev/sdb1 /
|
||||
|
||||
[exited with code 0]
|
||||
@@ -0,0 +1,8 @@
|
||||
felhom.eu -> 37.191.56.193 | 200 server=
|
||||
www.felhom.eu -> 37.191.56.193 | 200 server=
|
||||
hub.felhom.eu -> 37.191.56.193 | 302 server=
|
||||
iso.felhom.eu -> 2a06:98c1:3121::1 | 404 server=cloudflare
|
||||
gitea.dooplex.hu -> 37.191.56.193 | 200 server=
|
||||
=== where the installer script is fetched from (installer docs / website)
|
||||
https://iso.felhom.eu/felhom-installer-1.30.0-pve9.2-1.iso
|
||||
https://iso.felhom.eu/felhom-installer-1.30.0-pve9.2-1.iso.sha256
|
||||
@@ -0,0 +1,63 @@
|
||||
=== pods: runner / argocd / vaultwarden / hub / website / registry
|
||||
actualbudget-system/actualbudget-7fb44cd948-2xqjc Running
|
||||
argocd/argocd-application-controller-0 Running
|
||||
argocd/argocd-applicationset-controller-746fdcd449-s87sk Running
|
||||
argocd/argocd-server-b7bb9755c-s96pt Running
|
||||
felhom-system/contact-mailer-5bb869b85b-pqtjt Running
|
||||
felhom-system/felhom-webpage-7f5c7968b5-g62t6 Running
|
||||
felhom-system/filebrowser-59ff87cd88-c8qqm Running
|
||||
felhom-system/hub-676bcc45cc-gnbzp Running
|
||||
felhom-system/umami-d84d978f8-95rsv Running
|
||||
felhom-system/umami-db-5fd98f59c5-slhcw Running
|
||||
gitea-system/act-runner-84ff4656c4-t2857 Running
|
||||
gitea-system/gitea-5dbcbf7c7b-4j8qp Running
|
||||
vaultwarden-system/vaultwarden-7d967778cf-5mkr5 Running
|
||||
=== PVCs of those namespaces (name, size, storageclass)
|
||||
felhom-system/filebrowser-db 100Mi longhorn
|
||||
felhom-system/filebrowser-files 1Gi longhorn
|
||||
felhom-system/hub-data 4Gi longhorn
|
||||
felhom-system/umami-db-data 2Gi longhorn
|
||||
gitea-system/act-runner-data 5Gi longhorn
|
||||
gitea-system/gitea-data 50Gi longhorn
|
||||
vaultwarden-system/vaultwarden-data 5Gi longhorn
|
||||
=== ArgoCD applications (names, sync policy auto?)
|
||||
actualbudget auto={"enabled":true}
|
||||
admin-tools auto={"prune":true,"selfHeal":true}
|
||||
adventurelog auto={"enabled":true}
|
||||
arcade auto={"enabled":true}
|
||||
audiobookshelf auto={"enabled":true}
|
||||
authentik auto={"prune":true,"selfHeal":true}
|
||||
booking auto={"enabled":true}
|
||||
bookstack auto={"enabled":true}
|
||||
calibre auto={"enabled":true}
|
||||
cnpg-operator auto={"enabled":false,"prune":true,"selfHeal":true}
|
||||
code-server auto={"enabled":true}
|
||||
controller auto={"enabled":true}
|
||||
crafty auto={"enabled":true}
|
||||
database-system auto=
|
||||
external-dns-pihole auto={"enabled":true}
|
||||
felhom auto={"enabled":false}
|
||||
fileshare auto={"enabled":true}
|
||||
gitea auto=
|
||||
glance auto={"enabled":true}
|
||||
headlamp auto={"enabled":true}
|
||||
homepage auto={"enabled":true}
|
||||
immich auto={"enabled":true}
|
||||
infrastructure auto=
|
||||
jarr auto={"enabled":true,"prune":true,"selfHeal":true}
|
||||
jarr-nightly auto={"enabled":true,"prune":true,"selfHeal":true}
|
||||
kisfenyo auto={"enabled":true}
|
||||
monitoring auto=
|
||||
nextcloud auto={"enabled":true}
|
||||
office auto={"enabled":true}
|
||||
opengist auto={"enabled":true}
|
||||
orsi auto={"enabled":true}
|
||||
outline auto={"enabled":true}
|
||||
paperless auto={"enabled":true}
|
||||
pihole auto={"enabled":false}
|
||||
plantit auto={"enabled":true}
|
||||
plex auto={"enabled":true}
|
||||
privatebin auto={"enabled":true}
|
||||
root-apps auto={"prune":true,"selfHeal":true}
|
||||
servarr auto={"enabled":true}
|
||||
storage auto={"enabled":true}
|
||||
@@ -0,0 +1,4 @@
|
||||
2026-10-22T19:45:26Z real-ish | wg pub 3q1B4aHW… | latest_handshake=1792698270 (now 1792698326, diff -56 s) | prev key file: absent
|
||||
== clock set +2 h: 2026-10-22T21:45:26Z
|
||||
== a handshake under the ahead clock at 21:46:32 (epoch 1792705591)
|
||||
2026-10-22T21:46:32Z real-ish | wg pub 3q1B4aHW… | latest_handshake=1792705591 (now 1792705592, diff -1 s) | prev key file: absent
|
||||
@@ -0,0 +1,3 @@
|
||||
== clock back + chrony: 2026-10-22T19:46:43Z synced=yes
|
||||
2026-10-22T19:46:43Z real-ish | wg pub 3q1B4aHW… | latest_handshake=1792705591 (now 1792698403, diff 7188 s) | prev key file: absent
|
||||
tunnel ping: ok
|
||||
@@ -0,0 +1,15 @@
|
||||
2026-10-22T19:50:56Z real-ish | wg pub 3q1B4aHW… | latest_handshake=1792698580 (now 1792698657, diff -77 s) | prev key file: absent
|
||||
interface: wg-felhom
|
||||
public key: 3q1B4aHW…
|
||||
listening port: 50300
|
||||
|
||||
peer: f3d1ZI7/…
|
||||
endpoint: 167.233.158.164:443
|
||||
allowed ips: 10.77.0.1/32, 10.77.0.250/32
|
||||
latest handshake: 1 minute, 17 seconds ago
|
||||
transfer: 4.14 MiB received, 2.44 MiB sent
|
||||
persistent keepalive: every 25 seconds
|
||||
tunnel-ping-ok
|
||||
2026-10-22T23:46:09+02:00 night1023 felhom-agent[35760]: time=2026-10-22T23:46:09.963+02:00 level=WARN msg="wgtunnel: tunnel handshake stale but endpoint IP unchanged — endpoint may be down; not restarting" endpoint_ip=167.233.158.164
|
||||
2026-10-22T21:47:09+02:00 night1023 felhom-agent[35760]: time=2026-10-22T21:47:09.791+02:00 level=WARN msg="wgtunnel: the latest handshake is stamped in the future — this box's clock ran ahead, and the endpoint refuses its newer handshakes; re-keying if it s
|
||||
2026-10-22T21:50:09+02:00 night1023 felhom-agent[35760]: time=2026-10-22T21:50:09.778+02:00 level=INFO msg="wgtunnel: the handshake is no longer stamped in the future — no re-key needed"
|
||||
@@ -0,0 +1,5 @@
|
||||
-A INPUT -s 167.233.158.164/32 -p udp -m udp --sport 443 -m u32 --u32 "0x0>>0x16&0x3c@0x8=0x1000000" -m comment --comment d30-test -j DROP
|
||||
2026-10-22T19:52:01Z real-ish | wg pub 3q1B4aHW… | latest_handshake=1792698706 (now 1792698721, diff -15 s) | prev key file: absent
|
||||
== clock set +2 h: 2026-10-22T21:52:01Z
|
||||
== a handshake under the ahead clock at 21:54:50 (epoch 1792706086)
|
||||
2026-10-22T21:54:50Z real-ish | wg pub 3q1B4aHW… | latest_handshake=1792706086 (now 1792706090, diff -4 s) | prev key file: absent
|
||||
@@ -0,0 +1,3 @@
|
||||
== clock back + chrony: 2026-10-22T19:55:01Z synced=yes
|
||||
2026-10-22T19:55:01Z real-ish | wg pub 3q1B4aHW… | latest_handshake=1792706086 (now 1792698901, diff 7185 s) | prev key file: absent
|
||||
tunnel ping: ok
|
||||
@@ -0,0 +1,10 @@
|
||||
2026-10-22T20:10:49Z real-ish | wg pub 3q1B4aHW… | latest_handshake=1792706086 (now 1792699849, diff 6237 s) | prev key file: present
|
||||
tunnel ping: FAILS
|
||||
2026-10-22T22:10:09+02:00 night1023 felhom-agent[35760]: time=2026-10-22T22:10:09.972+02:00 level=WARN msg="wgtunnel: RE-KEYED after a handshake stamped in the future (D30) — registering the new key; the hub re-keys the peer in place and the endpoint takes the new key" old_pubkey="3q1B4aHW7EohL7Y8
|
||||
2026-10-22T22:10:10+02:00 night1023 felhom-agent[35760]: time=2026-10-22T22:10:10.772+02:00 level=INFO msg="wgtunnel: registered with hub" pubkey="HAYwc1CxA7Qb19sd6tlX1Y5YxHM8FeN0Tp8VCa2K8lY=" assigned_ip=10.77.0.5/32 existed=false generation=3 sync=ok
|
||||
total 20
|
||||
-rw------- last-applied.sha
|
||||
-rw------- private.key
|
||||
-rw------- private.key.prev
|
||||
-rw------- registered.json
|
||||
-rw------- wg-felhom.conf
|
||||
@@ -0,0 +1,9 @@
|
||||
2026-10-22T20:16:04Z real-ish | wg pub HAYwc1Cx… | latest_handshake=1792700136 (now 1792700164, diff -28 s) | prev key file: present
|
||||
tunnel ping: ok
|
||||
2026-10-22T22:10:09+02:00 night1023 felhom-agent[35760]: time=2026-10-22T22:10:09.972+02:00 level=WARN msg="wgtunnel: RE-KEYED after a handshake stamped in the future (D30) — registering the new key; the hub re-keys the peer in place and the endpoint takes the new key" old_pubkey="3q1B4aHW7EohL7Y8qbE5Z0lbo21kqT3mocvigZdPrBg="
|
||||
2026-10-22T22:10:10+02:00 night1023 felhom-agent[35760]: time=2026-10-22T22:10:10.772+02:00 level=INFO msg="wgtunnel: registered with hub" pubkey="HAYwc1CxA7Qb19sd6tlX1Y5YxHM8FeN0Tp8VCa2K8lY=" assigned_ip=10.77.0.5/32 existed=false generation=3 sync=ok
|
||||
2026-10-22T22:11:09+02:00 night1023 felhom-agent[35760]: time=2026-10-22T22:11:09.867+02:00 level=WARN msg="wgtunnel: the latest handshake is stamped in the future — this box's clock ran ahead, and the endpoint refuses its newer handshakes; re-keying if it stays" future_s=6217 rekey_after=15m0s
|
||||
2026-10-22T22:15:36+02:00 night1023 felhom-agent[35760]: time=2026-10-22T22:15:36.774+02:00 level=INFO msg="wgtunnel: tunnel conf applied" endpoint=167.233.158.164:443 dns_name=ep0.felhom.eu assigned_ip=10.77.0.5/32 action=restart
|
||||
2026-10-22T22:15:36+02:00 night1023 felhom-agent[35760]: time=2026-10-22T22:15:36.849+02:00 level=INFO msg="wgtunnel: the handshake is no longer stamped in the future — no re-key needed"
|
||||
(re-registers while the served block still named the old key, once a minute, existed=true: 5)
|
||||
rules left: 0
|
||||
@@ -0,0 +1,11 @@
|
||||
== hub DB read (named non-secret columns), host Tester4-16fdef
|
||||
wg_peers cols: ['pubkey', 'assigned_ip', 'host_id', 'note', 'created_at', 'updated_at']
|
||||
wg_peer ('Tester4-16fdef', 'HAYwc1Cx…', '10.77.0.5', '2026-10-22 20:10:10')
|
||||
host ('Tester4-16fdef', 3, '2026-10-22 20:15:36')
|
||||
host_reports cols: ['id', 'host_id', 'customer_id', 'received_at', 'report_json', 'agent_version', 'cpu_percent', 'memory_percent', 'disk_percent', 'guest_total', 'guest_running', 'cloudflared_status']
|
||||
2026-10-22 19:13:31 wg: {'pubkey': '', 'active': False, 'last_handshake_age_s': None, 'handshake_future_s': None}
|
||||
2026-10-22 19:28:36 wg: {'pubkey': '3q1B4aHW…', 'active': True, 'last_handshake_age_s': 14, 'handshake_future_s': None}
|
||||
2026-10-22 19:30:14 wg: {'pubkey': '3q1B4aHW…', 'active': True, 'last_handshake_age_s': 112, 'handshake_future_s': None}
|
||||
2026-10-22 19:45:20 wg: {'pubkey': '3q1B4aHW…', 'active': True, 'last_handshake_age_s': 50, 'handshake_future_s': None}
|
||||
2026-10-22 20:00:35 wg: {'pubkey': '3q1B4aHW…', 'active': True, 'last_handshake_age_s': 0, 'handshake_future_s': 6851}
|
||||
2026-10-22 20:15:36 wg: {'pubkey': 'HAYwc1Cx…', 'active': True, 'last_handshake_age_s': 0, 'handshake_future_s': 5950}
|
||||
@@ -0,0 +1,18 @@
|
||||
# Part 2 — D30 live: clock ahead → (alarm input) → re-key → handshake
|
||||
|
||||
Box: test box Tester4-16fdef (nested VM 9344 on demo-hp, installed tonight from ISO 1.30.0), agent fixture
|
||||
`0.163.0-night.1` = felhom-agent `ae69ad4` (D30). Hub: the live 0.165.0 (no alarm yet — the alarm and its clearing are
|
||||
pinned by hub tests, `84751cb0`, not deployed). Tool: `d30-box.sh` (in the scratchpad; state lines print 8 characters
|
||||
of a PUBLIC key, never a private key).
|
||||
|
||||
| File | What |
|
||||
|---|---|
|
||||
| `01a-run1-clock-ahead.txt`, `02a-run1-clock-back.txt`, `03a-run1-ep0-initiated-heal.txt` | Run 1: clock +2 h → a handshake stamped +2 h; clock back → the agent saw the future stamp (21:47) and, 3 min later, the stamp was in the past: **ep0 started a handshake of its own** (it had traffic for the box; the box accepts ep0's timestamps), the tunnel healed, the agent correctly did not re-key |
|
||||
| `04-run2-block-and-ahead.txt`, `05-run2-clock-back.txt` | Run 2, the rehearsal's condition: ep0's handshake starts dropped on the box (iptables `u32` on WireGuard message type 1, comment `d30-test`); clock +2 h, back |
|
||||
| `06-run2-after-rekey.txt` | the tunnel dead; at 22:10:09 local **RE-KEYED** (old `3q1B4aHW…` → new `HAYwc1Cx…`), the hub: `existed=false generation=3 sync=ok`, same 10.77.0.5; the old key kept as `private.key.prev` |
|
||||
| `07-run2-new-key-handshake.txt` | 22:15:36 local the desired state carried the new key → conf applied → **a handshake with the new key**, tunnel ping ok; the test rule removed (rules left: 0) |
|
||||
| `08-hub-db-read.txt` | the hub (named columns): `wg_peers` holds the new key for the host since 20:10:10Z; the stored reports show `handshake_future_s=6851` at 20:00:35Z (the alarm's input) |
|
||||
|
||||
Timeline (UTC): clock back 19:55:01 → future stamp seen 19:55:09 → re-key 20:10:09 → hub took it 20:10:10 → the box
|
||||
switched 20:15:36 (its next desired-state fetch) → handshake 20:15:36. Outage after the clock was right: ~20 min (the
|
||||
rehearsal's: until 2028 or a hand reset on ep0).
|
||||
@@ -0,0 +1,222 @@
|
||||
diff --git a/configs/felhom-os-apply b/configs/felhom-os-apply
|
||||
index 81565fb..ee8429f 100755
|
||||
--- a/configs/felhom-os-apply
|
||||
+++ b/configs/felhom-os-apply
|
||||
@@ -165,7 +165,8 @@ KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
|
||||
# dest: (source file under configs/, mode, check, policy)
|
||||
# check: the content check BEFORE anything is written (R18): sudoers → visudo -cf; sh / bash → -n; python → compile;
|
||||
# unit → a [Unit]/[Service]/[Timer] section; unit-nort → that, and never RuntimeDirectory= (the G1 incident);
|
||||
-# agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none.
|
||||
+# agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none;
|
||||
+# sshd → `sshd -t -f` on the snippet alone (R-982).
|
||||
# policy: replace → always written when it differs; if-absent → only when the box has none (a setting the operator may
|
||||
# have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none).
|
||||
# Order matters: the sudoers files come LAST, so a referenced wrapper is in place before the line that allows it.
|
||||
@@ -184,6 +185,9 @@ BUNDLE_KEEP = 3
|
||||
BUNDLE_MAX_BYTES = 4 * 1024 * 1024
|
||||
BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$")
|
||||
OOB_DIR = "/etc/felhom-sshd"
|
||||
+# R-982: the host's own sshd drop-in (after_install reloads sshd for it; undo reloads after putting it back).
|
||||
+SSHD_DROPIN = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
|
||||
+
|
||||
BUNDLE_FILES = [
|
||||
("/usr/local/sbin/felhom-mkfs-guarded", "felhom-mkfs-guarded.sh", 0o755, "bash", "replace"),
|
||||
("/usr/local/sbin/felhom-selfupdate-guarded", "felhom-selfupdate-guarded", 0o755, "sh", "replace"),
|
||||
@@ -221,6 +225,9 @@ BUNDLE_FILES = [
|
||||
("/etc/systemd/system/felhom-sshd.service", "felhom-sshd.service", 0o644, "unit-nort", "oob"),
|
||||
("/etc/felhom-oob.nft", "felhom-oob.nft", 0o644, "nft", "oob"),
|
||||
("/etc/systemd/system/felhom-oob-nft.service", "felhom-oob-nft.service", 0o644, "unit", "oob"),
|
||||
+ # R-982, operator ruling D4 (2026-10-22): the host's own sshd (port 22, the LAN): root by key only. The console
|
||||
+ # password stays; felhom-sshd (the operator's door) reads nothing from sshd_config.d. Reloaded in after_install.
|
||||
+ ("/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf", "felhom-root-ssh-key-only.conf", 0o644, "sshd", "replace"),
|
||||
("/etc/sudoers.d/felhom-op", "felhom-op.sudoers", 0o440, "sudoers", "oob"),
|
||||
("/etc/sudoers.d/felhom-agent", "felhom-agent.sudoers", 0o440, "sudoers", "replace"),
|
||||
]
|
||||
@@ -434,7 +441,8 @@ class Runner:
|
||||
def check_content(self, kind, data):
|
||||
"""Run an external syntax check on a root-only temp copy. Returns (rc, message)."""
|
||||
import tempfile
|
||||
- cmd = {"sudoers": ["visudo", "-cf"], "sh": ["sh", "-n"], "bash": ["bash", "-n"], "nft": ["nft", "-c", "-f"]}[kind]
|
||||
+ cmd = {"sudoers": ["visudo", "-cf"], "sh": ["sh", "-n"], "bash": ["bash", "-n"], "nft": ["nft", "-c", "-f"],
|
||||
+ "sshd": ["/usr/sbin/sshd", "-t", "-f"]}[kind]
|
||||
with tempfile.TemporaryDirectory(prefix="felhom-bundle-") as d:
|
||||
p = os.path.join(d, "f")
|
||||
with open(p, "wb") as f:
|
||||
@@ -1864,6 +1872,7 @@ class Bundle:
|
||||
def __init__(self, apply):
|
||||
self.a, self.r = apply, apply.r
|
||||
self.report = apply.report
|
||||
+ self.sshd_dropin_not_in_force = False
|
||||
|
||||
# ---------- reading and checking ----------
|
||||
def load_file(self, path):
|
||||
@@ -1932,10 +1941,17 @@ class Bundle:
|
||||
text = content.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
raise Refused("R18", f"{dest} is not UTF-8 text")
|
||||
- if check in ("sudoers", "sh", "bash", "nft"):
|
||||
+ if check in ("sudoers", "sh", "bash", "nft", "sshd"):
|
||||
rc, msg = self.r.check_content(check, content)
|
||||
if rc != 0:
|
||||
raise Refused("R18", f"{dest} fails its {check} check: {msg}")
|
||||
+ if check == "sshd":
|
||||
+ # R-982: the drop-in's PermitRootLogin must be exactly `prohibit-password`. `no` would also shut the root key
|
||||
+ # out (Proxmox's own node-to-node SSH and the LAN key route), `yes` would undo the ruling — both refused.
|
||||
+ # Only that line is pinned: the file is signed like every bundle file (review A7).
|
||||
+ vals = re.findall(r"^\s*PermitRootLogin\s+(\S+)\s*$", text, re.M | re.I)
|
||||
+ if vals != ["prohibit-password"]:
|
||||
+ raise Refused("R18", f"{dest} must set exactly `PermitRootLogin prohibit-password` (found {vals})")
|
||||
elif check == "python":
|
||||
try:
|
||||
compile(text, dest, "exec")
|
||||
@@ -2044,6 +2060,8 @@ class Bundle:
|
||||
self.report["failed"] = {"rc": 3, "step": "bundle", "reason": reason[:300]}
|
||||
return 3
|
||||
rep["signers_created"] = False
|
||||
+ if self.sshd_dropin_not_in_force:
|
||||
+ rep["sshd_dropin_not_in_force"] = True # R-982 A2: written and reloaded, but sshd -T shows root passwords allowed
|
||||
if bootstrap and not self.r.lexists(TRUST_SIGNERS):
|
||||
self.r.put_file(TRUST_SIGNERS, pinned_signers_line().encode(), 0o644)
|
||||
rep["signers_created"] = True
|
||||
@@ -2073,6 +2091,23 @@ class Bundle:
|
||||
"enable the crash guard")
|
||||
if "/etc/systemd/system/felhom-mgmt-watchdog.timer" in dests:
|
||||
self.must(["systemctl", "enable", "--now", "felhom-mgmt-watchdog.timer"], "enable the mgmt watchdog timer")
|
||||
+ # R-982: the host's sshd takes the root-key-only drop-in. The WHOLE config is tested first (a failure is a
|
||||
+ # refusal, and the bundle undo puts the old files back); `try-reload-or-restart` keeps every open session and
|
||||
+ # does nothing on a box whose ssh.service is not running (a socket-activated or stopped sshd reads the file
|
||||
+ # at its next start) — a failure there must not freeze the bundle route (review 2026-10-22 night, A1).
|
||||
+ # felhom-sshd is not touched. Then the POSITIVE check: sshd's own effective view for root. A box whose
|
||||
+ # sshd_config has no Include of sshd_config.d, an earlier drop-in or a Match block can keep the password route
|
||||
+ # — that is said loudly in the log and the report, never silently recorded as done (A2).
|
||||
+ if SSHD_DROPIN in written:
|
||||
+ self.must(["/usr/sbin/sshd", "-t"], "sshd -t (the host's whole sshd config)")
|
||||
+ self.must(["systemctl", "try-reload-or-restart", "ssh"], "systemctl try-reload-or-restart ssh")
|
||||
+ rc, out, err = self.r.host(["/usr/sbin/sshd", "-T", "-C", "user=root,host=felhom-check,addr=192.0.2.1"], 60)
|
||||
+ eff = re.findall(r"^permitrootlogin\s+(\S+)", out, re.M | re.I)
|
||||
+ if rc != 0 or eff != ["without-password"]:
|
||||
+ self.r.log(f"os-apply: BUNDLE WARNING the root-key-only drop-in is NOT in force: sshd -T says "
|
||||
+ f"permitrootlogin={eff or '?'} (rc={rc}) — check sshd_config's Include, an earlier drop-in "
|
||||
+ f"or a Match block (R-982)")
|
||||
+ self.sshd_dropin_not_in_force = True
|
||||
|
||||
def must(self, argv, what):
|
||||
rc, out, err = self.r.host(argv, 120)
|
||||
@@ -2133,6 +2168,9 @@ class Bundle:
|
||||
self.r.log(f"os-apply: BUNDLE UNDO could not restore {dest}: {e}")
|
||||
if any(d.startswith("/etc/systemd/system/") for d in back):
|
||||
self.r.host(["systemctl", "daemon-reload"], 120)
|
||||
+ if SSHD_DROPIN in back:
|
||||
+ # A3: after_install may already have reloaded sshd with the drop-in; the restored state takes effect now.
|
||||
+ self.r.host(["systemctl", "try-reload-or-restart", "ssh"], 120)
|
||||
rc, _, _ = self.r.host(["visudo", "-c"], 60)
|
||||
self.r.log(f"os-apply: BUNDLE UNDO restored={len(back)} visudo-after={'ok' if rc == 0 else 'FAILED'}")
|
||||
return back
|
||||
diff --git a/configs/felhom-root-ssh-key-only.conf b/configs/felhom-root-ssh-key-only.conf
|
||||
new file mode 100644
|
||||
index 0000000..d92eea9
|
||||
--- /dev/null
|
||||
+++ b/configs/felhom-root-ssh-key-only.conf
|
||||
@@ -0,0 +1,7 @@
|
||||
+# MANAGED BY FELHOM (the agent's config bundle) — R-982, operator ruling D4 (2026-10-22).
|
||||
+# The Proxmox host's OWN sshd — the listener the household network reaches on port 22: root logs in with a KEY only.
|
||||
+# The root password still works on the console (the break-glass password: felhom.eu runbooks/break-glass.md). The
|
||||
+# operator's door (felhom-sshd, port 8822, /etc/felhom-sshd/sshd_config) is a separate daemon whose config includes
|
||||
+# nothing from here, so it is not changed. Debian's sshd_config reads this folder BEFORE its own settings, and for
|
||||
+# sshd the first value wins; the bundle checks the effective value with `sshd -T` after the reload.
|
||||
+PermitRootLogin prohibit-password
|
||||
diff --git a/configs/test_felhom_config_bundle.py b/configs/test_felhom_config_bundle.py
|
||||
index 1c5a6ab..58b5238 100644
|
||||
--- a/configs/test_felhom_config_bundle.py
|
||||
+++ b/configs/test_felhom_config_bundle.py
|
||||
@@ -429,6 +429,90 @@ class SelfCheckUndo(unittest.TestCase):
|
||||
self.assert_restored(box, before, rep)
|
||||
|
||||
|
||||
+
|
||||
+class RootKeyOnly(unittest.TestCase):
|
||||
+ """R-982, operator ruling D4 (2026-10-22): the bundle gives the host's own sshd a root-key-only drop-in, tests the
|
||||
+ whole config and reloads sshd — only when the file was written; a drop-in that says anything but
|
||||
+ `prohibit-password` is refused before anything is written.
|
||||
+ RED-PROOF: with the after_install block removed, test_fresh_box_takes_the_dropin_and_reloads fails (no reload);
|
||||
+ with the content rule removed, test_a_dropin_that_says_yes_or_no_is_refused fails — seen failing 2026-10-22."""
|
||||
+ DEST = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
|
||||
+
|
||||
+ def test_fresh_box_takes_the_dropin_and_reloads(self):
|
||||
+ data, sha = real_bundle()
|
||||
+ box = Box(data, signed_job(sha))
|
||||
+ rc, rep = run(box)
|
||||
+ self.assertEqual(rc, 0, rep)
|
||||
+ self.assertIn(b"PermitRootLogin prohibit-password", box.files[self.DEST])
|
||||
+ self.assertEqual(box.modes[self.DEST], 0o644)
|
||||
+ self.assertIn(["/usr/sbin/sshd", "-t"], box.calls)
|
||||
+ self.assertIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
|
||||
+ self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
|
||||
+ self.assertFalse(any("felhom-sshd" in " ".join(c) for c in box.calls), "the operator's door must not be touched")
|
||||
+
|
||||
+ def test_an_unchanged_dropin_is_not_reloaded(self):
|
||||
+ data, sha = real_bundle()
|
||||
+ box = Box(data, signed_job(sha))
|
||||
+ box.put(self.DEST, (REPO / "configs" / "felhom-root-ssh-key-only.conf").read_bytes(), 0o644)
|
||||
+ rc, rep = run(box)
|
||||
+ self.assertEqual(rc, 0, rep)
|
||||
+ self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
|
||||
+
|
||||
+ def test_a_dropin_that_says_yes_or_no_is_refused(self):
|
||||
+ for bad in (b"PermitRootLogin yes\n", b"PermitRootLogin no\n", b"# nothing\n",
|
||||
+ b"PermitRootLogin prohibit-password\nPermitRootLogin yes\n"):
|
||||
+ def edit(files, bad=bad):
|
||||
+ for e in files:
|
||||
+ if e["path"] == self.DEST:
|
||||
+ e["content_b64"] = base64.b64encode(bad).decode()
|
||||
+ data, sha = edited_bundle(edit)
|
||||
+ box = Box(data, signed_job(sha))
|
||||
+ rc, rep = run(box)
|
||||
+ self.assertNotEqual(rc, 0, bad)
|
||||
+ self.assertNotIn(self.DEST, box.files, bad)
|
||||
+
|
||||
+ # Review 2026-10-22 night. A2: the effective value is read back with `sshd -T`; a drop-in that is not in force is
|
||||
+ # said in the log and the report (not refused — a refusal would freeze the bundle route). A3: a bundle undone
|
||||
+ # after the reload reloads sshd again with the old state. RED-PROOF: with the sshd -T check removed,
|
||||
+ # test_a_dropin_not_in_force_is_reported fails; with the undo reload removed, test_an_undone_bundle_reloads_sshd
|
||||
+ # fails — seen failing 2026-10-22 night.
|
||||
+ def _sshd_T(self, box, value):
|
||||
+ orig = box.host
|
||||
+ def host(argv, timeout=600, stdin=None):
|
||||
+ if argv[:2] == ["/usr/sbin/sshd", "-T"]:
|
||||
+ box.calls.append(argv)
|
||||
+ return 0, f"port 22\npermitrootlogin {value}\npasswordauthentication yes\n", ""
|
||||
+ return orig(argv, timeout, stdin)
|
||||
+ box.host = host
|
||||
+
|
||||
+ def test_the_effective_value_is_checked(self):
|
||||
+ data, sha = real_bundle()
|
||||
+ box = Box(data, signed_job(sha))
|
||||
+ self._sshd_T(box, "without-password")
|
||||
+ rc, rep = run(box)
|
||||
+ self.assertEqual(rc, 0, rep)
|
||||
+ self.assertTrue(any(c[:2] == ["/usr/sbin/sshd", "-T"] for c in box.calls))
|
||||
+ self.assertNotIn("sshd_dropin_not_in_force", rep["bundle"])
|
||||
+
|
||||
+ def test_a_dropin_not_in_force_is_reported(self):
|
||||
+ data, sha = real_bundle()
|
||||
+ box = Box(data, signed_job(sha))
|
||||
+ self._sshd_T(box, "yes") # e.g. no Include of sshd_config.d
|
||||
+ rc, rep = run(box)
|
||||
+ self.assertEqual(rc, 0, rep)
|
||||
+ self.assertTrue(rep["bundle"].get("sshd_dropin_not_in_force"), rep)
|
||||
+ self.assertTrue(any("NOT in force" in l for l in box.logs), box.logs)
|
||||
+
|
||||
+ def test_an_undone_bundle_reloads_sshd(self):
|
||||
+ data, sha = real_bundle()
|
||||
+ box = Box(data, signed_job(sha))
|
||||
+ box.visudo_fail = True # the self-check fails AFTER the install → undo
|
||||
+ rc, rep = run(box)
|
||||
+ self.assertEqual(rc, 3, rep)
|
||||
+ self.assertNotIn(self.DEST, box.files)
|
||||
+ reloads = [i for i, c in enumerate(box.calls) if c == ["systemctl", "try-reload-or-restart", "ssh"]]
|
||||
+ self.assertEqual(len(reloads), 2, box.calls) # after_install, then the undo
|
||||
+
|
||||
class TrustBootstrap(unittest.TestCase):
|
||||
def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self):
|
||||
data, sha = real_bundle()
|
||||
@@ -0,0 +1,29 @@
|
||||
diff --git a/scripts/felhom-host-install.sh b/scripts/felhom-host-install.sh
|
||||
index 970109fd..349da0c5 100644
|
||||
--- a/scripts/felhom-host-install.sh
|
||||
+++ b/scripts/felhom-host-install.sh
|
||||
@@ -1417,6 +1417,12 @@ run_uninstall() {
|
||||
if [[ -f /etc/default/grub.d/zz-felhom-kernel-default.cfg ]]; then
|
||||
log_info " kept /etc/default/grub.d/zz-felhom-kernel-default.cfg (the kernel this box booted healthily stays the default)"
|
||||
fi
|
||||
+ # R-982 (operator ruling D4, 2026-10-22; agent bundle after v0.162.0): the host sshd's root-key-only drop-in. The
|
||||
+ # uninstall gives the host back as Proxmox set it up, so the drop-in goes and sshd is reloaded (open sessions stay).
|
||||
+ if [[ -f /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf ]]; then
|
||||
+ run rm -f /etc/ssh/sshd_config.d/10-felhom-root-key-only.conf
|
||||
+ run systemctl try-reload-or-restart ssh 2>/dev/null || log_warn " sshd was not reloaded — the drop-in is gone, sshd reads it at its next start (R-982)"
|
||||
+ fi
|
||||
# 1.31.0 (R-840): the bundle's previous copies and the wrapper's nonce record.
|
||||
if [[ -d /var/lib/felhom-os-apply ]]; then run rm -rf /var/lib/felhom-os-apply; fi
|
||||
if [[ -f /var/lib/vz/snippets/felhom-guest-hook.sh ]]; then run rm -f /var/lib/vz/snippets/felhom-guest-hook.sh; fi
|
||||
diff --git a/scripts/test_hostinstall.py b/scripts/test_hostinstall.py
|
||||
index 3f7088a6..e9cf9ad9 100644
|
||||
--- a/scripts/test_hostinstall.py
|
||||
+++ b/scripts/test_hostinstall.py
|
||||
@@ -60,6 +60,7 @@ BUNDLE_DESTS = [
|
||||
"/etc/systemd/system/felhom-sshd.service",
|
||||
"/etc/felhom-oob.nft",
|
||||
"/etc/systemd/system/felhom-oob-nft.service",
|
||||
+ "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf", # R-982, operator ruling D4 (2026-10-22)
|
||||
"/etc/sudoers.d/felhom-op",
|
||||
"/etc/sudoers.d/felhom-agent",
|
||||
]
|
||||
@@ -0,0 +1,4 @@
|
||||
GET HTTP/2 200
|
||||
fields: ['pairing_code', 'passphrase', 'robots', 'viewport']
|
||||
POST HTTP/2 200
|
||||
answer: Felhom — Doboz összekötése Felhom doboz összekötése Sikeres összekötés. A doboz kb. egy percen belül folytatja a telepítést. Ezt az oldalt bezárhatod — a beállítás a háttérben befejeződik, és a vezérlőpultod hamarosan elérhető lesz. Felhom.eu
|
||||
@@ -0,0 +1,3 @@
|
||||
GET /claim HTTP/2 200
|
||||
csrf_len 64 cookie_present yes
|
||||
POST /claim HTTP/2 302 location: /
|
||||
@@ -0,0 +1,17 @@
|
||||
== 19:30:07 agent fixture
|
||||
felhom-agent 0.163.0-night.1
|
||||
active
|
||||
ac39dadd162bc74f
|
||||
== 19:30:11 D4 drop-in by hand (what the bundle after_install does: sshd -t -f, install, sshd -t, try-reload-or-restart, sshd -T)
|
||||
snippet check rc=0
|
||||
sshd -t ok
|
||||
reloaded
|
||||
permitrootlogin without-password
|
||||
passwordauthentication yes
|
||||
== 19:30:28 D4 checks from DooPlex (LAN)
|
||||
root PASSWORD over SSH: Permission denied, please try again.
|
||||
root KEY over SSH: root
|
||||
web page password login (/api2/json/access/ticket): HTTP 200
|
||||
== 19:30:43 controller fixture
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.340.0-rc.6 Up 15 seconds (healthy)
|
||||
2026/10/22 19:30:45 [INFO] [tier2] start-up: no leftover second-drive copy container
|
||||
@@ -0,0 +1,10 @@
|
||||
== 19:32:04 POST /api/storage/init device=/dev/sdb fstype=ext4 mount_name=hdd_1
|
||||
200
|
||||
{
|
||||
"data": {
|
||||
"phase": "formatting",
|
||||
"started": true
|
||||
},
|
||||
"ok": true
|
||||
}
|
||||
200{"data":{"device":"/dev/sdb","durable_id":"","error":"","opsign":"","phase":"done","reason":"","started_at":"2026-10-22T19:32:09.922168725Z","updated_at":"2026-10-22T19:32:12.640993967Z","where":"/mnt/felhom-drives/hdd_1"},"ok":true}
|
||||
@@ -0,0 +1,17 @@
|
||||
19:32:39Z nextcloud deploy 202 Telepítés elindítva – az állapot a kártyán követhető
|
||||
19:34:09Z nextcloud → running 97 s
|
||||
19:34:10Z immich deploy 202 Telepítés elindítva – az állapot a kártyán követhető
|
||||
19:36:31Z immich → running 141 s
|
||||
19:36:31Z paperless-ngx deploy 202 Telepítés elindítva – az állapot a kártyán követhető
|
||||
19:38:41Z paperless-ngx → running 130 s
|
||||
19:38:41Z vaultwarden deploy 202 Telepítés elindítva – az állapot a kártyán követhető
|
||||
19:39:01Z vaultwarden → running 20 s
|
||||
19:39:02Z mealie deploy 202 Telepítés elindítva – az állapot a kártyán követhető
|
||||
19:40:32Z mealie → running 90 s
|
||||
19:40:33Z actualbudget deploy 202 Telepítés elindítva – az állapot a kártyán követhető
|
||||
19:41:03Z actualbudget → running 30 s
|
||||
19:41:03Z radicale deploy 202 Telepítés elindítva – az állapot a kártyán követhető
|
||||
19:41:33Z radicale → running 30 s
|
||||
19:41:33Z bookstack deploy 202 Telepítés elindítva – az állapot a kártyán követhető
|
||||
19:42:43Z bookstack → running 70 s
|
||||
19:42:43Z memory: {"data": {"vmid": 9201, "allocated_mb": 4096, "usage_mb": 2126, "host_total_mb": 7939, "min_mb": 2048, "max_mb": 5891, "floor_mb": 2638, "running": true, "host_swap_mb": 7935, "guest_swap_mb": 4096}, "ok": true}
|
||||
@@ -0,0 +1,7 @@
|
||||
WARN: Systemd 257 detected. You may need to enable nesting.
|
||||
Task finished with 1 warning(s)!
|
||||
hostname: night1023-loader
|
||||
memory: 2048
|
||||
rootfs: nvme-scratch:9345/vm-9345-disk-0.raw,size=24G
|
||||
192.168.0.152
|
||||
19:39:56
|
||||
@@ -0,0 +1,22 @@
|
||||
Formatting '/mnt/hdd_1/images/9344/vm-9344-disk-0.raw', fmt=raw size=274877906944 preallocation=off
|
||||
scsi0: successfully created disk 'nvme-scratch:9344/vm-9344-disk-0.raw,size=256G'
|
||||
Formatting '/mnt/hdd_1/images/9344/vm-9344-disk-1.raw', fmt=raw size=214748364800 preallocation=off
|
||||
scsi1: successfully created disk 'nvme-scratch:9344/vm-9344-disk-1.raw,size=200G'
|
||||
update VM 9344: -boot order=scsi0;ide2
|
||||
18:46:17
|
||||
boot: order=scsi0;ide2
|
||||
cores: 4
|
||||
cpu: host
|
||||
ide2: local:iso/felhom-installer-1.30.0-pve9.2-1.iso,media=cdrom,size=1665356K
|
||||
memory: 8192
|
||||
meta: creation-qemu=11.0.3,ctime=1792694775
|
||||
name: night1023-soak
|
||||
net0: virtio=BC:24:11:AB:3A:21,bridge=vmbr0
|
||||
ostype: l26
|
||||
scsi0: nvme-scratch:9344/vm-9344-disk-0.raw,size=256G
|
||||
scsi1: nvme-scratch:9344/vm-9344-disk-1.raw,size=200G
|
||||
scsihw: virtio-scsi-single
|
||||
smbios1: uuid=cbc4fbdd-86b6-4050-9f10-7f4319c9f86b
|
||||
vga: std
|
||||
vmgenid: aa1b5848-cd99-43d0-ae66-f222698ee0ab
|
||||
Install pressed 18:51:58
|
||||
@@ -0,0 +1,4 @@
|
||||
bind POST 19:12:17
|
||||
claim 19:29:40
|
||||
env written: IRISZ ODON COOKIE
|
||||
soak started 19:55:59
|
||||
@@ -69,7 +69,7 @@ local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom
|
||||
7. **Gitea:** `gitea-restore.md` (on a normal network this time). Then rebuild the images from the code into the new
|
||||
registry: hub, controller, agent (`RUNBOOK-manual-build.md`).
|
||||
8. **The hub:** k3s, then `RUNBOOK-hub-db-offsite-backup.md` §3 with S2 and S4 — **start it with mail held**
|
||||
(§3 step 1: the `MAIL-HOLD` marker, from the next hub release; until then: no network until the pending notices are understood).
|
||||
(§3 step 1: the `MAIL-HOLD` marker — shipped in hub 0.145.0, 2026-10-10).
|
||||
9. **DNS:** in Cloudflare (S11) point `hub.felhom.eu` (today a CNAME to `dooplex.hopto.org`) and `gitea.dooplex.hu` at
|
||||
the new place. The boxes reconnect by themselves: their API keys are in the restored hub DB.
|
||||
10. **Signing:** the keys are in the restored copy (`signing/`) and on the sheet (S6) — put them back and sign as before. Without it, no box accepts an agent update or a
|
||||
|
||||
Reference in New Issue
Block a user