morning after: R-889 delivered (controller v0.299.0, df's percent read back), R-776/R-613 proven on 9202 and live, R-585/R-621 delivered; ten operator questions on one page in STATUS; report (164 -> 150; 0 opened, 14 closed)
gates / gates (push) Successful in 2m27s
gates / gates (push) Successful in 2m27s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -16,6 +16,14 @@
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **2026-10-06 (morning) — the morning after (rulings `09` §3 137–138).** Register 164 → 150 (0 opened, 14 closed).
|
||||
> Installer 1.32.0 PUBLISHED (tag `installer-v1.32.0` = `32a1520833`, pins `dbede797`; public sha = tag). R-889: every
|
||||
> disk percent is `df`'s (`system.DFUsedPercent`, controller `bee2c2d`) — controller v0.299.0 (`ff4a99a`, MinAgent 0.131.0)
|
||||
> + golden 0.299.0 (`9948aa16…`, pinned Docker set), vouched with agent 0.148.0, floors 0.299.0 for the three boxes;
|
||||
> read back demo-hp SSD 21 % → 23 % = df. Catalog `3896cb0`: R-776 (kimai/zipline/vikunja/nextcloud trust the docker
|
||||
> networks) + R-613 (nextcloud probe `"installed":true`), each proven on 9202 with a control. Scratch 9202's controller
|
||||
> now 0.299.0 (set by hand). Ten operator questions: `STATUS.md`. Report: `REPORT-morning-after-2026-10-06.md`.
|
||||
|
||||
> **2026-10-06 (night) — the burn-down night (releases).** Register 199 → 164 (1 opened: R-889 disk percent vs `df`;
|
||||
> 36 closed). Releases in window 1 (00:30–00:58): hub v0.138.0 (manifest `edde9e13`; R-879 seal — 16 legacy values sealed,
|
||||
> raw DB copy 0 plaintext; roll-back = `felhom-hub -unseal-box-secrets` in the pod first), agent v0.148.0 (tag = `861d32a`,
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
# REPORT — the morning after: installer 1.32.0 published, the real disk percentage (R-889), the four held catalog fixes proven, ten questions on one page — 2026-10-06
|
||||
|
||||
| Part | Result |
|
||||
|---|---|
|
||||
| **A** — publish installer 1.32.0 | **done** — tag `installer-v1.32.0` (= `32a1520833`), both `webpage.yaml` pins moved (`dbede797`), synced; the public URL serves `SCRIPT_VERSION="1.32.0"`, sha256 `70a4ea9f…` = the tag's file; 9 installer rows closed |
|
||||
| **B** — the real disk percentage (R-889) | **done** — cause measured (the root reserve was in the denominator); one function `DFUsedPercent` for every disk percent; controller **v0.299.0** + golden 0.299.0 delivered to demo-hp, demo-felhom, Tester 1; read back: demo-hp SSD 21 % → 23 % = `df` |
|
||||
| **C** — the four waiting catalog fixes | **done** — all proven on scratch 9202 with a control each, pushed to the live catalog (`3896cb0`); R-776 and R-613 closed |
|
||||
| **D** — the ten questions on one page | **done** — `STATUS.md` „Ten questions for you", each with two options, the cost, what happens if nothing, and a pick |
|
||||
|
||||
| Rows before | Rows after | Opened | Closed |
|
||||
|---|---|---|---|
|
||||
| **164** | **150** | **0** | **14** |
|
||||
|
||||
Counted by `register_shape_gate.py`'s method. Closed: R-275, R-276, R-881, R-306, R-130, R-180, R-179, R-310, R-274 (installer),
|
||||
R-889, R-776, R-613, R-585, R-621.
|
||||
|
||||
## Baselines and rulings
|
||||
|
||||
Verified 07:49: felhom.eu `32a1520833` · hub v0.138.0 · agent v0.148.0 · controller v0.298.0 (+ unreleased) · register 164.
|
||||
The two rulings were recorded first as `09` §3 decisions 137 (publish 1.32.0) and 138 (R-889: `df`'s number, alarm levels kept).
|
||||
|
||||
## Part A — the installer
|
||||
|
||||
Tag → push (pre-push gates OK) → both pins → commit `dbede797` → ArgoCD sync → `felhom-webpage` rolled out. The first
|
||||
read-back hit a short 502 during the pod swap (the downloaded file was an nginx error page); the second, with `curl -f`,
|
||||
read 1.32.0 and the tag's exact bytes (`audits/morning-after-2026-10-06/installer-publish.txt`). CI 1404 (tag) and 1405
|
||||
(pins) green.
|
||||
|
||||
## Part B — R-889
|
||||
|
||||
**Measured, not guessed** (demo-hp 9201, `stat -f` + `df -P` on `/mnt/sys_drive`): blocks 18016108, free 14150899, avail
|
||||
13229302 → the old `(blocks − free) / blocks` = 21.5 %, `df` = 23 %. The gap is the 5 % reserved for root, which `df`
|
||||
leaves out of the denominator. Fix: `system.DFUsedPercent(used, avail)` = used / (used + avail), used by `GetDiskUsage`,
|
||||
`readDiskUsage`, the recovery-unit headroom projection and the deploy page's free percent (`bee2c2d`). The tile's „(/)"
|
||||
label measured the docker data volume, so it now says „Rendszer" / „System" (7 parity fixtures changed by exactly those
|
||||
bytes). Tests: `TestDFUsedPercent_MatchesDFOnRealNumbers` (the demo-hp numbers and the R-516 shape) and
|
||||
`TestR889_EveryDiskPercentIsTheDFOne` (a source scan); red-proved by putting the old line back. Full suite rc 0, gates rc 0.
|
||||
|
||||
Release v0.299.0 (`ff4a99a`, MinAgent 0.131.0; also carries R-585, R-621, R-516 from the night), image built, **golden
|
||||
0.299.0 baked with the pinned Docker set from the start** (sha `9948aa16…`, round trip equal, token leak 0 with a working
|
||||
control, VM back on `virgin`), vouched (agent 0.148.0, min_agent 0.131.0), floors 0.299.0 for the three boxes only.
|
||||
Delivered by 06:17:33Z. **Read back** (the hub page, from each box's report, vs `df` in the guest — two channels): demo-hp
|
||||
SSD **23 %** (df 23 %; was 21 %), NVMe 6 % (df 7 %; df rounds up), demo-felhom 1 % (df 1 %). Tester 1 reports 0.299.0; its
|
||||
`df` was not read (no direct access). CI 1406 green.
|
||||
|
||||
## Part C — the four catalog fixes on 9202
|
||||
|
||||
Method (`audits/morning-after-2026-10-06/live-9202/README.md`): 9202's controller set by hand to 0.299.0; the drill repo
|
||||
reset to live `d955df1` + the five held commits; 9202 pointed at it per `09` §6.5; every request through the simulated
|
||||
tunnel (a curl container at cloudflared's address), the stranger forging a new leftmost `X-Forwarded-For` each try; each
|
||||
app also run as a **control** with its new line removed from the stack's compose and restarted through the product.
|
||||
|
||||
| App | With the fix | Control |
|
||||
|---|---|---|
|
||||
| vikunja | stranger 10 × 403 then 429; household **200** | household **429** |
|
||||
| zipline | stranger 7 × 400 then 429; household **200** | household **429** |
|
||||
| kimai | stranger 5 × wrong then THROTTLED; household **ok** | household **THROTTLED** |
|
||||
| nextcloud | brute-force attempts: visitor **6**, forged/tunnel/traefik 0 | visitor **0** |
|
||||
| nextcloud probe | `status.php` `{"installed":true,…}`, controller `running` | — |
|
||||
|
||||
**Not established:** where nextcloud's control tries were counted (its throttle lives in Redis; traefik's two addresses
|
||||
read 0). Controls held: demo-hp's catalog cache and the live `main` stayed `d955df1` during the test. **The stall of last
|
||||
night was not repeated:** this time the lead ran the proof itself, so there was no helper to stall. Teardown in the
|
||||
README; the catalog pointer was restored byte-identical (`cmp`).
|
||||
|
||||
**Found on the way, fixed or noted:** the walk tool's default data path (`<drive>/userdata/<app>`) is a folder inside a
|
||||
drive, which R-839's new refusal rejects — passed the drive itself instead (the tool still carries the old default; it is
|
||||
only reached on 9202). One `docker ps` of mine was filtered by a „perl" locale filter that also dropped the `paperless-*`
|
||||
lines; paperless had run since 00:30:09Z, before the work — said in the README so nobody reads it as a start caused here.
|
||||
|
||||
## Part D
|
||||
|
||||
`STATUS.md`, „Ten questions for you": R-444, R-99, R-618, R-645, R-856, R-747, R-734, R-624, R-502, R-774. A reply of
|
||||
„all as picked" is enough.
|
||||
|
||||
## CI, last commit of every repo
|
||||
|
||||
felhom.eu: this commit (its run is checked by its commit after the push; the previous one, `e6cfe6d`, is 1407 success); felhom-controller `ff4a99a` → 1406 success;
|
||||
app-catalog `3896cb0` → 1408 success; felhom-agent `37e98f4` (unchanged) → 1399 success.
|
||||
|
||||
## Teardown
|
||||
|
||||
Drill VM: build guest destroyed, secrets shredded, off, `virgin`. 9202: four throwaway apps removed through the product,
|
||||
their own folders removed by name, the catalog pointer restored byte-identical, tools and password files deleted; its
|
||||
controller stays on 0.299.0. demo-hp host: nothing changed. Hub: the vouch and three floors. Scratch secrets shredded.
|
||||
@@ -1,11 +1,32 @@
|
||||
# STATUS — what works, what's broken, what's next
|
||||
|
||||
**Ready for the first real tester (Tester-2): yes. Tester 2 (a laptop, off at night) was offline again; nothing was
|
||||
sent to it.**
|
||||
**Ready for the first real tester (Tester-2): yes. Tester 2 (a laptop) is off; nothing was sent to it.**
|
||||
|
||||
**Updated 2026-10-06 06:00 (the burn-down night): every box of ours healthy on hub 0.138.0, agent 0.148.0, controller
|
||||
0.298.0. The open-items list is at 164 (was 199). Morning note: `documentation/audits/night-burndown-2026-10-05/MORNING-NOTE.md`;
|
||||
report: `REPORT-burndown3-2026-10-06.md`.**
|
||||
**Updated 2026-10-06 09:00: every box of ours healthy on hub 0.138.0, agent 0.148.0, controller 0.299.0; installer 1.32.0
|
||||
is public. The open-items list is at 150. Report: `REPORT-morning-after-2026-10-06.md`.**
|
||||
|
||||
## This morning (2026-10-06): your two answers done; the list at 150
|
||||
|
||||
- **Installer 1.32.0 is published.** The public script reads 1.32.0 and is byte-identical to its tag.
|
||||
- **The box shows the real disk use** (the number `df` gives). demo-hp's SSD went from 21 % to 23 %, the same as `df`.
|
||||
Alarm levels are unchanged, so a full disk alarms a little earlier. Released as controller 0.299.0 to all three boxes.
|
||||
- **The four waiting app fixes passed on the scratch box and are live** (visitor addresses for kimai, zipline, vikunja,
|
||||
nextcloud; nextcloud's health check). Each was tested against a control.
|
||||
|
||||
## Ten questions for you — answer „all as picked", or name the ones you want differently
|
||||
|
||||
| # | Question | Option A | Option B | If you decide nothing | My pick |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | **R-444** — Should the boxes trim their customer guests' disks once a week, so deleted data stops filling the disk pool? | Yes: one new admin permission (`pct fstrim`), weekly, outside the night, measured once on demo-hp first | No: leave it; the pool keeps space nobody uses | Nothing changes; a full pool can one day stop every guest on that box | **A** |
|
||||
| 2 | **R-99** — Should broken leftovers of aborted backups be deleted on the backup server? | Yes, by hand, by a runbook, when one is seen | No: they are detected, harmless, and do not affect what is kept | They stay; one small leftover per aborted upload | **B** |
|
||||
| 3 | **R-618** — May an app update count Docker's own „healthy" as proof that the new version works? | No: keep our own check only | Yes: Docker's healthy can end a wait early | Nothing changes (A) | **A** |
|
||||
| 4 | **R-645** — When you lift a held update by hand, how do we stop the night backup from saving the broken version over the good copy? | The night backup skips an app whose saved version is not what it runs | Lifting a hold by hand is refused; you use „Undo" instead | The good copy can be overwritten within seconds of a hand lift | **A** |
|
||||
| 5 | **R-856** — After a crash restart, should app mails wait longer (about 15 minutes), so the household gets one message, not three? | Yes: a longer quiet time after a crash boot (two repositories change) | No: close it; one crash can send several true mails | Several mails after a crash, as now | **A** |
|
||||
| 6 | **R-747** — Should mealie's app page tell the household that five wrong logins lock the account for 1–2 hours? | Yes: one sentence on the page | No | A locked household does not know why or for how long | **A** |
|
||||
| 7 | **R-734** — Should the update test ignore small marker files an app rewrites at every start (immich)? | Yes: a per-app list of such files, each with a reason | No: keep marking it; immich updates then need a fresh full copy first | immich's night update is skipped where no fresh copy exists | **A** |
|
||||
| 8 | **R-624** — May the update test hold an app's admin password to create test data in apps that refuse strangers (vaultwarden, zipline)? | Yes, on the test bench only | No: write down that these apps are tested by hand | No change; those two apps stay hand-tested | **B** |
|
||||
| 9 | **R-502** — May a slow check that starts Docker containers run on DooPlex (the ISO's first-boot test)? | Yes, in full runs only (never on every push), with a clear „not checked" when Docker is missing | No: it stays a hand step of each ISO release | It stays a hand step; it can be forgotten | **B** |
|
||||
| 10 | **R-774** — Should Karakeep's page say that its phone app sends crash reports to its makers? | Yes: one sentence on the page | No | The household is not told | **A** |
|
||||
|
||||
## Tonight (2026-10-06): the list at 164
|
||||
|
||||
@@ -14,7 +35,7 @@ hub 0.138.0 (secrets in the hub database are sealed; checked on a copy: none rea
|
||||
files), controller 0.298.0 and a new install image 0.298.0. The app catalog was updated. Six small decisions were taken
|
||||
by CC unattended (`09` §3 decisions 131–136); each can be reversed.
|
||||
|
||||
**Needs you (none urgent; if you do nothing, each stays as it is):**
|
||||
**Needs you (none urgent; if you do nothing, each stays as it is):** *(items 1 and 2 answered 2026-10-06 07:45 and done; item 3 is the table above)*
|
||||
1. **Publish installer 1.32.0?** Nine uninstall/pre-flight fixes wait on `main`. If nothing: new installs keep the old
|
||||
installer. Pick: yes.
|
||||
2. **R-889 — the disk percentage** reads ~5 points low (not `df`'s formula), so fill alarms come late. Pick: use `df`'s
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
== controller delivery 2026-10-06T06:47:49Z
|
||||
demo-hp + demo-felhom: gitea.dooplex.hu/admin/felhom-controller:0.299.0 (healthy) at 06:17:33Z (docker ps)
|
||||
tester-1: hub customer page 'Controller version 0.299.0', 'Controller elindult (0.299.0)'
|
||||
@@ -0,0 +1,12 @@
|
||||
== before: hub customer page demo-hp 'SSD 21% 14.7 / 68.7 GB' (df in the guest: /mnt/sys_drive 23%)
|
||||
== vouch 2026-10-06T06:16:27Z: agent 0.148.0, golden 0.299.0, min_agent 0.131.0
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /configuration?flash=artifacts_set
|
||||
== floors 2026-10-06T06:16:56Z: min_controller_version=0.299.0 min_agent=0.131.0
|
||||
demo-hp: Location: /customers/demo-hp?flash=floor_set
|
||||
demo-felhom: Location: /customers/demo-felhom?flash=floor_set
|
||||
tester-1: Location: /customers/tester-1?flash=floor_set
|
||||
2026/10/06 08:16:56 [INFO] Artifact manifest set: agent=0.148.0 golden=0.299.0 min_agent="0.131.0" wrapper_sha=false bundle_sha="a6fa4f589d184b58c9911303bd087e300be1e75b3647e4302c9594df6989c4de"
|
||||
2026/10/06 08:16:56 [INFO] Customer demo-hp controller-version floor override set to "0.299.0" (declared MinAgent "0.131.0")
|
||||
2026/10/06 08:16:56 [INFO] Customer demo-felhom controller-version floor override set to "0.299.0" (declared MinAgent "0.131.0")
|
||||
2026/10/06 08:16:56 [INFO] Customer tester-1 controller-version floor override set to "0.299.0" (declared MinAgent "0.131.0")
|
||||
@@ -0,0 +1,28 @@
|
||||
# Scratch 9202 — the four held catalog fixes proven (2026-10-06 morning)
|
||||
|
||||
Venue: scratch guest 9202 on demo-hp, controller set BY HAND to 0.299.0 (was 0.296.0; allowed only there), pointed at the
|
||||
drill catalog `admin/app-catalog-drill` = live catalog `d955df1` + the five held commits (`c5674ce`), by `tools/repoint.py`
|
||||
(`09` §6.5: saved copy `controller.yaml.pre-morning1006`, cache removed, restart). Controls: demo-hp's 9201 cache and the
|
||||
live catalog's `main` both stayed `d955df1`. Every request went through the SIMULATED tunnel: a curl container at
|
||||
172.16.253.2 (cloudflared's address, the only one traefik trusts for forwarded headers) sending
|
||||
`X-Forwarded-For: <forged>, <visitor>` — the stranger changed the forged leftmost entry on every try.
|
||||
|
||||
| App (row) | With the fix | Control: the line removed from the stack's compose, restart through the product |
|
||||
|---|---|---|
|
||||
| nextcloud (R-776) | 6 wrong WebDAV logins → `occ security:bruteforce:attempts`: **visitor 6**, every forged address 0, the tunnel 0, traefik 0 | visitor **0** (traefik 0 too: this nextcloud keeps its throttle in Redis, so where the control's tries were counted was not found) |
|
||||
| nextcloud (R-613) | `status.php` = `{"installed":true,"maintenance":false,…}`; the controller's state `running` with the new probe | — |
|
||||
| vikunja (R-776) | stranger: 10 × 403 then 429; **household 200** | stranger the same; **household 429** |
|
||||
| zipline (R-776) | stranger: 7 × 400 then 429; **household 200** | stranger the same; **household 429** |
|
||||
| kimai (R-776) | stranger: 5 × wrong then THROTTLED; **household ok** | stranger the same; **household THROTTLED** |
|
||||
|
||||
So with the fix each app throttles the VISITOR, a stranger cannot escape by forging the leftmost address, and the
|
||||
household is not locked out by a stranger; without it the household is.
|
||||
|
||||
**Teardown:** each app removed through the product (keep drive data, drop backups), its own folders under the scratch
|
||||
drive removed by name (`teardown.txt`); `controller.yaml` restored byte-identical (`cmp`); the tools and password files
|
||||
deleted in the guest. 9202's controller stays on 0.299.0 (newer than before; `/etc/felhom-controller-image.pre-morning1006`
|
||||
holds the old line). The drill repo keeps the tested branch. Host demo-hp and the hub: nothing changed.
|
||||
|
||||
**Said plainly:** a log filter used during this work drops lines containing „perl" (locale noise) — it also dropped the
|
||||
`paperless-*` containers from one `docker ps`, which made it look as if paperless started during the work. It did not:
|
||||
it has run since 2026-10-06 00:30:09Z (`docker inspect` StartedAt), before any of this.
|
||||
@@ -0,0 +1,16 @@
|
||||
##### kimai on 9202 — controller gitea.dooplex.hu/admin/felhom-controller:0.299.0
|
||||
deploy -> True
|
||||
env: ['TRUSTED_PROXIES=127.0.0.1,172.16.0.0/12']
|
||||
control first: the household logs in at once -> ok
|
||||
## WITH the fix (TRUSTED_PROXIES=127.0.0.1,172.16.0.0/12) — 2026-10-06T06:41:42Z
|
||||
stranger 198.51.100.66, 7 wrong for admin@example.com (a new forged leftmost each): ['wrong', 'wrong', 'wrong', 'wrong', 'wrong', 'THROTTLED', 'THROTTLED']
|
||||
household 203.0.113.10, the right password, at once: ok
|
||||
## CONTROL: the line removed 0
|
||||
restart -> 200
|
||||
env: ['TRUSTED_PROXIES=nginx,localhost,127.0.0.1']
|
||||
## CONTROL — the template before R-776 — 2026-10-06T06:42:52Z
|
||||
stranger 198.51.100.66, 7 wrong for admin@example.com (a new forged leftmost each): ['wrong', 'wrong', 'wrong', 'wrong', 'wrong', 'THROTTLED', 'THROTTLED']
|
||||
household 203.0.113.10, the right password, at once: THROTTLED
|
||||
## put the template's compose back: 1
|
||||
restart -> 200
|
||||
env back: ['TRUSTED_PROXIES=127.0.0.1,172.16.0.0/12']
|
||||
@@ -0,0 +1,33 @@
|
||||
##### nextcloud on 9202 — controller gitea.dooplex.hu/admin/felhom-controller:0.299.0 — 2026-10-06T06:21:57Z
|
||||
deploy -> True
|
||||
the container env: 172.16.0.0/12
|
||||
R-613 status.php bytes (inside the container): {"installed":true,"maintenance":false,"needsDbUpgrade":false,"version":"34.0.4.1","versionstring":"34.0.4","edition":"","productname":"Nextcloud","extendedSupport":false}
|
||||
R-613 the controller's state for the app (the probe expects '"installed":true'): running | health:
|
||||
trusted_proxies in config.php: 172.16.0.0/12
|
||||
stranger 198.51.100.66, 6 wrong basic-auth tries with a new forged leftmost each: [['401'], ['401'], ['401'], ['401'], ['401'], ['401']]
|
||||
occ bruteforce:attempts 198.51.100.66 (the real visitor): - bypass-listed: false - attempts: 6 - delay: 6400
|
||||
occ bruteforce:attempts 10.5.0.1 (forged / the tunnel): - bypass-listed: false - attempts: 0 - delay: 0
|
||||
occ bruteforce:attempts 10.5.0.6 (forged / the tunnel): - bypass-listed: false - attempts: 0 - delay: 0
|
||||
occ bruteforce:attempts 172.16.253.2 (forged / the tunnel): - bypass-listed: false - attempts: 0 - delay: 0
|
||||
occ bruteforce:attempts 172.16.253.3 (traefik): - bypass-listed: false - attempts: 0 - delay: 0
|
||||
occ bruteforce:attempts 172.18.0.5 (traefik): - bypass-listed: false - attempts: 0 - delay: 0
|
||||
occ bruteforce:attempts 203.0.113.10 (another visitor, never failed): - bypass-listed: false - attempts: 0 - delay: 0
|
||||
## CONTROL — trusted_proxies removed by hand (the template before R-776) — 2026-10-06T06:24:58Z
|
||||
delete: System config value trusted_proxies deleted | now: 172.16.0.0/12
|
||||
stranger 198.51.100.77, 3 wrong tries: [['401'], ['401'], ['401']]
|
||||
attempts 198.51.100.77 (visitor): - bypass-listed: false - attempts: 3 - delay: 800
|
||||
attempts 172.16.253.3 (traefik): - bypass-listed: false - attempts: 0 - delay: 0
|
||||
attempts 172.18.0.5 (traefik): - bypass-listed: false - attempts: 0 - delay: 0
|
||||
restore: System config value trusted_proxies => 0 set to string 172.16.0.0/12 | now: 172.16.0.0/12
|
||||
## CONTROL 2 — TRUSTED_PROXIES removed from the stack's compose (= the template before R-776) — 2026-10-06T06:26:13Z
|
||||
0
|
||||
System config value trusted_proxies deleted
|
||||
restart through the product -> 200
|
||||
env now: (unset) | trusted_proxies:
|
||||
stranger 198.51.100.88, 3 wrong tries: [['401'], ['401'], ['401']]
|
||||
attempts 198.51.100.88 (visitor): - bypass-listed: false - attempts: 0 - delay: 0
|
||||
attempts 172.16.253.3 (traefik): - bypass-listed: false - attempts: 0 - delay: 0
|
||||
attempts 172.18.0.5 (traefik): - bypass-listed: false - attempts: 0 - delay: 0
|
||||
## put the template's compose back: 1
|
||||
restart through the product -> 200
|
||||
env back: 172.16.0.0/12
|
||||
@@ -0,0 +1,91 @@
|
||||
nextcloud: stop -> 200
|
||||
nextcloud: remove (keep drive data, drop backups) -> 200 {'ok': True, 'data': {'removed': 'nextcloud', 'volumes_removed': ['nextcloud_nextcloud_db_data', 'nextcloud_nextcloud_ht
|
||||
nextcloud: tidy own folders by name: removed /mnt/felhom-drives/scratch_hdd/appdata/nextcloud
|
||||
removed /mnt/felhom-drives/scratch_hdd/userdata/nextcloud
|
||||
nextcloud: after: deployed=False leftovers='/opt/docker/stacks/nextcloud'
|
||||
vikunja: stop -> 200
|
||||
vikunja: remove (keep drive data, drop backups) -> 200 {'ok': True, 'data': {'removed': 'vikunja', 'volumes_removed': ['vikunja_vikunja_data', 'vikunja_vikunja_db'], 'hdd_path
|
||||
vikunja: tidy own folders by name:
|
||||
vikunja: after: deployed=False leftovers='/opt/docker/stacks/vikunja'
|
||||
zipline: stop -> 200
|
||||
zipline: remove (keep drive data, drop backups) -> 200 {'ok': True, 'data': {'removed': 'zipline', 'volumes_removed': ['zipline_zipline_postgres_data', 'zipline_zipline_public
|
||||
zipline: tidy own folders by name:
|
||||
zipline: after: deployed=False leftovers='/opt/docker/stacks/zipline'
|
||||
kimai: stop -> 200
|
||||
kimai: remove (keep drive data, drop backups) -> 200 {'ok': True, 'data': {'removed': 'kimai', 'volumes_removed': ['kimai_kimai_db_data', 'kimai_kimai_var'], 'hdd_paths_remo
|
||||
kimai: tidy own folders by name:
|
||||
kimai: after: deployed=False leftovers='/opt/docker/stacks/kimai'
|
||||
26: repo_url: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git
|
||||
3
|
||||
RESTORED-IDENTICAL
|
||||
|
||||
0
|
||||
filebrowser
|
||||
felhom-controller
|
||||
paperless-webserver
|
||||
paperless-postgres
|
||||
paperless-redis
|
||||
traefik
|
||||
actualbudget
|
||||
adventurelog
|
||||
audiobookshelf
|
||||
bentopdf
|
||||
bookstack
|
||||
calcom
|
||||
calibre-web
|
||||
chaoscrash
|
||||
chaosoom
|
||||
chaosoomb
|
||||
claper
|
||||
code-server
|
||||
crafty-controller
|
||||
dawarich
|
||||
docmost
|
||||
emby
|
||||
filebrowser
|
||||
ghost
|
||||
gitea
|
||||
glance
|
||||
gokapi
|
||||
grafana
|
||||
gramps-web
|
||||
grimmory
|
||||
home-assistant
|
||||
homebox
|
||||
homepage
|
||||
immich
|
||||
jellyfin
|
||||
karakeep
|
||||
kimai
|
||||
komga
|
||||
mealie
|
||||
metube
|
||||
n8n
|
||||
navidrome
|
||||
nextcloud
|
||||
onlyoffice
|
||||
opengist
|
||||
outline
|
||||
paperless-ngx
|
||||
papra
|
||||
plant-it
|
||||
plex
|
||||
privatebin
|
||||
radarr
|
||||
radicale
|
||||
rallly
|
||||
recipe-importer
|
||||
romm
|
||||
seerr
|
||||
sonarr
|
||||
sparkyfitness
|
||||
tandoor
|
||||
termix
|
||||
traefik
|
||||
uptime-kuma
|
||||
vaultwarden
|
||||
vikunja
|
||||
wanderer
|
||||
wger
|
||||
wishlist
|
||||
zipline
|
||||
@@ -0,0 +1,10 @@
|
||||
#!/bin/sh
|
||||
# burst.sh <host> <path> <n> <bad-body> <good-body-file> — n wrong POSTs from stranger 198.51.100.66 (a new forged
|
||||
# leftmost each), then ONE right POST from household 203.0.113.10, all through the simulated tunnel, back to back.
|
||||
H=$1; P=$2; N=$3; BAD=$4; GOOD=$5; out=""
|
||||
i=1; while [ $i -le $N ]; do
|
||||
c=$(curl -sk -o /dev/null -w '%{http_code}' -X POST "https://traefik$P" -H "Host: $H" -H "X-Forwarded-For: 10.5.0.$i, 198.51.100.66" \
|
||||
-H "CF-Connecting-IP: 198.51.100.66" -H "Content-Type: application/json" --data "$BAD"); out="$out $c"; i=$((i+1)); done
|
||||
g=$(curl -sk -o /dev/null -w '%{http_code}' -X POST "https://traefik$P" -H "Host: $H" -H "X-Forwarded-For: 203.0.113.10" \
|
||||
-H "CF-Connecting-IP: 203.0.113.10" -H "Content-Type: application/json" --data @"$GOOD")
|
||||
echo "stranger:$out | household: $g"
|
||||
@@ -0,0 +1,34 @@
|
||||
import sys, time
|
||||
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
|
||||
import box_walk as w
|
||||
APP, SUB, EMAIL = "kimai", "time", "admin@example.com"
|
||||
EVF = open(f"{w.EV}/kimai.txt", "a", buffering=1)
|
||||
def say(*a):
|
||||
w.say(*a); EVF.write(" ".join(map(str, a)) + "\n")
|
||||
def login(v, f, pw):
|
||||
return w.guest(f"docker run --rm --network felhom-tunnel --ip 172.16.253.2 -v /root/kmlogin.sh:/s.sh:ro -v /root/.kmpw:/pw:ro "
|
||||
f"curlimages/curl:8.11.1 sh /s.sh {v} {f} {EMAIL} {pw}").strip().splitlines()[-1]
|
||||
def rounds(label):
|
||||
say(f"## {label} — {time.strftime('%FT%TZ', time.gmtime())}")
|
||||
say(" stranger 198.51.100.66, 7 wrong for", EMAIL, "(a new forged leftmost each):", [login("198.51.100.66", f"10.5.0.{i}", "-wrong") for i in range(1, 8)])
|
||||
say(" household 203.0.113.10, the right password, at once:", login("203.0.113.10", "203.0.113.10", "/pw"))
|
||||
w.login()
|
||||
say(f"##### kimai on 9202 — controller {w.guest('docker inspect felhom-controller --format {{.Config.Image}}').strip()}")
|
||||
say("deploy ->", w.deploy(APP, SUB))
|
||||
pw = (w.GENERATED.get(APP) or {}).get("ADMIN_PASSWORD")
|
||||
if not pw: sys.exit("no generated ADMIN_PASSWORD")
|
||||
w.guest(f"umask 077; printf '%s' '{pw}' > /root/.kmpw; chmod 644 /root/.kmpw")
|
||||
w.wait_app(SUB, "/en/login", tries=60)
|
||||
env = lambda: [l for l in w.guest("docker inspect kimai --format '{{range .Config.Env}}{{println .}}{{end}}'").splitlines() if "TRUSTED_PROXIES" in l]
|
||||
say("env:", env())
|
||||
say("control first: the household logs in at once ->", login("203.0.113.10", "203.0.113.10", "/pw"))
|
||||
rounds("WITH the fix (TRUSTED_PROXIES=127.0.0.1,172.16.0.0/12)")
|
||||
D = "/opt/docker/stacks/kimai"
|
||||
say("## CONTROL: the line removed", w.guest(f"cp -p {D}/docker-compose.yml /root/km.with; sed -i '/- TRUSTED_PROXIES=/d' {D}/docker-compose.yml; grep -c TRUSTED_PROXIES {D}/docker-compose.yml").strip())
|
||||
c, d = w.ctl("POST", f"/api/stacks/{APP}/restart"); say(" restart ->", c); time.sleep(15); w.wait_app(SUB, "/en/login", tries=60)
|
||||
say(" env:", env())
|
||||
rounds("CONTROL — the template before R-776")
|
||||
say("## put the template's compose back:", w.guest(f"cp -p /root/km.with {D}/docker-compose.yml; grep -c TRUSTED_PROXIES {D}/docker-compose.yml").strip())
|
||||
c, d = w.ctl("POST", f"/api/stacks/{APP}/restart"); say(" restart ->", c); time.sleep(15)
|
||||
say(" env back:", env())
|
||||
w.guest("rm -f /root/.kmpw")
|
||||
@@ -0,0 +1,14 @@
|
||||
#!/bin/sh
|
||||
# kmlogin.sh <visitor> <forged-left> <email> <pwfile|-wrong> — ONE Kimai form login through the SIMULATED tunnel.
|
||||
# Prints: ok | wrong | THROTTLED | ?<code>
|
||||
V=$1; F=$2; E=$3; P=$4; H=time.enkisfelhom.hu; J=/tmp/jar.$$
|
||||
x="X-Forwarded-For: $F, $V"; c="CF-Connecting-IP: $V"
|
||||
curl -sk -c $J -b $J "https://traefik/en/login" -H "Host: $H" -H "$x" -H "$c" -o /tmp/p.$$
|
||||
T=$(grep -o 'name="_csrf_token" value="[^"]*"' /tmp/p.$$ | head -1 | sed 's/.*value="//;s/"$//')
|
||||
if [ "$P" = "-wrong" ]; then PW="wrong-$$"; else PW=$(cat "$P"); fi
|
||||
L=$(curl -sk -c $J -b $J -o /dev/null -w '%{http_code} %{redirect_url}' "https://traefik/en/login_check" -H "Host: $H" -H "$x" -H "$c" \
|
||||
--data-urlencode "_csrf_token=$T" --data-urlencode "_username=$E" --data-urlencode "_password=$PW")
|
||||
case "$L" in *"/login"*) curl -sk -c $J -b $J "https://traefik/en/login" -H "Host: $H" -H "$x" -H "$c" -o /tmp/q.$$
|
||||
if grep -qiE 'too many|try again in' /tmp/q.$$; then echo THROTTLED; else echo wrong; fi;;
|
||||
"302 "*) echo ok;; *) echo "?$L";; esac
|
||||
rm -f $J /tmp/p.$$ /tmp/q.$$
|
||||
@@ -0,0 +1,40 @@
|
||||
import sys, time, json
|
||||
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
|
||||
import box_walk as w
|
||||
EVF = open(f"{w.EV}/nextcloud.txt", "a", buffering=1)
|
||||
def say(*a):
|
||||
w.say(*a); EVF.write(" ".join(map(str, a)) + "\n")
|
||||
APP, SUB = "nextcloud", "cloud"
|
||||
H = f"{SUB}.{w.DOMAIN}"
|
||||
def tun(v, f, m, p, b="", ct="", x=""):
|
||||
q = lambda s: "'" + s.replace("'", "'\\''") + "'"
|
||||
return w.guest(f"docker run --rm --network felhom-tunnel --ip 172.16.253.2 -v /root/tun.sh:/t.sh:ro curlimages/curl:8.11.1 "
|
||||
f"sh /t.sh {H} {v} {f} {m} {q(p)} {q(b)} {q(ct)} {q(x)}").strip().splitlines()[-1:]
|
||||
w.login()
|
||||
say(f"##### nextcloud on 9202 — controller {w.guest('docker inspect felhom-controller --format {{.Config.Image}}').strip()} — {time.strftime('%FT%TZ', time.gmtime())}")
|
||||
say("deploy ->", w.deploy(APP, SUB, {"HDD_PATH": "/mnt/felhom-drives/scratch_hdd"}))
|
||||
say("the container env:", w.guest("docker exec nextcloud printenv TRUSTED_PROXIES").strip())
|
||||
# R-613: wait until the app is up, then read status.php's exact bytes and the controller's own verdict
|
||||
for i in range(90):
|
||||
st = w.guest("docker exec nextcloud curl -s http://localhost/status.php").strip()
|
||||
if '"installed":true' in st: break
|
||||
time.sleep(10)
|
||||
say("R-613 status.php bytes (inside the container):", st[:200])
|
||||
for i in range(30):
|
||||
s = w.stack(APP); state = s.get("state")
|
||||
if state == "running": break
|
||||
time.sleep(10)
|
||||
say("R-613 the controller's state for the app (the probe expects '\"installed\":true'):", state, "| health:", (s.get("health") or s.get("health_status") or ""))
|
||||
say("trusted_proxies in config.php:", w.guest("docker exec -u www-data nextcloud php occ config:system:get trusted_proxies 2>&1 | tr '\\n' ' '").strip())
|
||||
# R-776 3.6: a stranger fails WebDAV basic auth 6 times through the simulated tunnel, each with a NEW forged leftmost
|
||||
V, LAN = "198.51.100.66", "203.0.113.10"
|
||||
codes = [tun(V, f"10.5.0.{i}", "PROPFIND", "/remote.php/dav/files/nobody/", "", "", "Authorization: Basic bm9ib2R5Ondyb25n") for i in range(1, 7)]
|
||||
say(f"stranger {V}, 6 wrong basic-auth tries with a new forged leftmost each:", codes)
|
||||
time.sleep(3)
|
||||
occ = lambda ip: w.guest(f"docker exec -u www-data nextcloud php occ security:bruteforce:attempts {ip} 2>&1 | tr '\\n' ' '").strip()
|
||||
say(f"occ bruteforce:attempts {V} (the real visitor):", occ(V))
|
||||
for f in ("10.5.0.1", "10.5.0.6", "172.16.253.2"):
|
||||
say(f"occ bruteforce:attempts {f} (forged / the tunnel):", occ(f))
|
||||
tip = w.guest("docker inspect traefik --format '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}'").split()
|
||||
for ip in tip: say(f"occ bruteforce:attempts {ip} (traefik):", occ(ip))
|
||||
say(f"occ bruteforce:attempts {LAN} (another visitor, never failed):", occ(LAN))
|
||||
@@ -0,0 +1,21 @@
|
||||
import sys, time
|
||||
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
|
||||
import box_walk as w
|
||||
EVF = open(f"{w.EV}/nextcloud.txt", "a", buffering=1)
|
||||
def say(*a):
|
||||
w.say(*a); EVF.write(" ".join(map(str, a)) + "\n")
|
||||
H = "cloud.enkisfelhom.hu"
|
||||
def tun(v, f, m, p, x=""):
|
||||
q = lambda s: "'" + s.replace("'", "'\\''") + "'"
|
||||
return w.guest(f"docker run --rm --network felhom-tunnel --ip 172.16.253.2 -v /root/tun.sh:/t.sh:ro curlimages/curl:8.11.1 "
|
||||
f"sh /t.sh {H} {v} {f} {m} {q(p)} '' '' {q(x)}").strip().splitlines()[-1:]
|
||||
occ = lambda a: w.guest(f"docker exec -u www-data nextcloud php occ {a} 2>&1 | tr '\\n' ' '").strip()
|
||||
say(f"## CONTROL — trusted_proxies removed by hand (the template before R-776) — {time.strftime('%FT%TZ', time.gmtime())}")
|
||||
say("delete:", occ("config:system:delete trusted_proxies"), "| now:", occ("config:system:get trusted_proxies"))
|
||||
V2 = "198.51.100.77"
|
||||
say(f"stranger {V2}, 3 wrong tries:", [tun(V2, f"10.6.0.{i}", "PROPFIND", "/remote.php/dav/files/nobody/", "Authorization: Basic bm9ib2R5Ondyb25n") for i in range(1, 4)])
|
||||
time.sleep(3)
|
||||
say(f"attempts {V2} (visitor):", occ(f"security:bruteforce:attempts {V2}"))
|
||||
for ip in w.guest("docker inspect traefik --format '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}'").split():
|
||||
say(f"attempts {ip} (traefik):", occ(f"security:bruteforce:attempts {ip}"))
|
||||
say("restore:", occ("config:system:set trusted_proxies 0 --value=172.16.0.0/12"), "| now:", occ("config:system:get trusted_proxies"))
|
||||
@@ -0,0 +1,33 @@
|
||||
import sys, time
|
||||
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
|
||||
import box_walk as w
|
||||
EVF = open(f"{w.EV}/nextcloud.txt", "a", buffering=1)
|
||||
def say(*a):
|
||||
w.say(*a); EVF.write(" ".join(map(str, a)) + "\n")
|
||||
H = "cloud.enkisfelhom.hu"
|
||||
def tun(v, f, m, p, x=""):
|
||||
q = lambda s: "'" + s.replace("'", "'\\''") + "'"
|
||||
return w.guest(f"docker run --rm --network felhom-tunnel --ip 172.16.253.2 -v /root/tun.sh:/t.sh:ro curlimages/curl:8.11.1 "
|
||||
f"sh /t.sh {H} {v} {f} {m} {q(p)} '' '' {q(x)}").strip().splitlines()[-1:]
|
||||
occ = lambda a: w.guest(f"docker exec -u www-data nextcloud php occ {a} 2>&1 | tr '\\n' ' '").strip()
|
||||
w.login()
|
||||
D = "/opt/docker/stacks/nextcloud"
|
||||
say(f"## CONTROL 2 — TRUSTED_PROXIES removed from the stack's compose (= the template before R-776) — {time.strftime('%FT%TZ', time.gmtime())}")
|
||||
say(w.guest(f"cd {D} && cp -p docker-compose.yml /root/nc-compose.with && sed -i '/- TRUSTED_PROXIES=/d' docker-compose.yml && grep -c TRUSTED_PROXIES docker-compose.yml; docker exec -u www-data nextcloud php occ config:system:delete trusted_proxies").strip())
|
||||
code, d = w.ctl("POST", "/api/stacks/nextcloud/restart"); say("restart through the product ->", code)
|
||||
for _ in range(40):
|
||||
if '"installed":true' in w.guest("docker exec nextcloud curl -s http://localhost/status.php"): break
|
||||
time.sleep(5)
|
||||
say("env now:", w.guest("docker exec nextcloud printenv TRUSTED_PROXIES || echo '(unset)'").strip(), "| trusted_proxies:", occ("config:system:get trusted_proxies"))
|
||||
V2 = "198.51.100.88"
|
||||
say(f"stranger {V2}, 3 wrong tries:", [tun(V2, f"10.7.0.{i}", "PROPFIND", "/remote.php/dav/files/nobody/", "Authorization: Basic bm9ib2R5Ondyb25n") for i in range(1, 4)])
|
||||
time.sleep(3)
|
||||
say(f"attempts {V2} (visitor):", occ(f"security:bruteforce:attempts {V2}"))
|
||||
for ip in w.guest("docker inspect traefik --format '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}'").split():
|
||||
say(f"attempts {ip} (traefik):", occ(f"security:bruteforce:attempts {ip}"))
|
||||
say("## put the template's compose back:", w.guest(f"cp -p /root/nc-compose.with {D}/docker-compose.yml && grep -c TRUSTED_PROXIES {D}/docker-compose.yml").strip())
|
||||
code, d = w.ctl("POST", "/api/stacks/nextcloud/restart"); say("restart through the product ->", code)
|
||||
for _ in range(40):
|
||||
if '"installed":true' in w.guest("docker exec nextcloud curl -s http://localhost/status.php"): break
|
||||
time.sleep(5)
|
||||
say("env back:", w.guest("docker exec nextcloud printenv TRUSTED_PROXIES").strip())
|
||||
@@ -0,0 +1,37 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Point 9202 at the drill catalog, or put the saved controller.yaml back. `09` §6.5."""
|
||||
import io, os, re, sys
|
||||
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
|
||||
import box_walk as w
|
||||
VOL = "/var/lib/docker/volumes/felhom-controller-data/_data"
|
||||
SAVE = f"{VOL}/controller.yaml.pre-morning1006"
|
||||
DRILL = "https://gitea.dooplex.hu/admin/app-catalog-drill.git"
|
||||
def creds():
|
||||
for l in io.open(os.path.expanduser("~/.git-credentials")).read().split("\n"):
|
||||
m = re.match(r"https://(admin):([^@]+)@gitea\.dooplex\.hu", l)
|
||||
if m: return m.group(1), m.group(2)
|
||||
sys.exit("no admin credential")
|
||||
if sys.argv[1] == "drill":
|
||||
u, t = creds()
|
||||
out = w.guest(f"""set -e
|
||||
test -f {SAVE} || cp -p {VOL}/controller.yaml {SAVE}
|
||||
python3 - <<'PY'
|
||||
import re
|
||||
p = "{VOL}/controller.yaml"; s = open(p).read()
|
||||
s = re.sub(r'(^\\s+repo_url: ).*$', r'\\g<1>{DRILL}', s, count=1, flags=re.M)
|
||||
s = re.sub(r'(^git:(?:\\n\\s+.*)*?\\n\\s+token: ).*$', r'\\g<1>"{t}"', s, count=1, flags=re.M)
|
||||
s = re.sub(r'(^git:(?:\\n\\s+.*)*?\\n\\s+username: ).*$', r'\\g<1>"{u}"', s, count=1, flags=re.M)
|
||||
open(p, "w").write(s)
|
||||
PY
|
||||
rm -rf {VOL}/catalog-cache {VOL}/data/catalog-cache
|
||||
docker restart felhom-controller >/dev/null
|
||||
grep -n 'repo_url' {VOL}/controller.yaml
|
||||
""")
|
||||
print(out.replace(t, "<token>"))
|
||||
elif sys.argv[1] == "restore":
|
||||
print(w.guest(f"""set -e
|
||||
cp -p {SAVE} {VOL}/controller.yaml
|
||||
rm -rf {VOL}/catalog-cache {VOL}/data/catalog-cache
|
||||
docker restart felhom-controller >/dev/null
|
||||
grep -n 'repo_url' {VOL}/controller.yaml; grep -c 'token: ""' {VOL}/controller.yaml || true
|
||||
cmp {SAVE} {VOL}/controller.yaml && echo RESTORED-IDENTICAL"""))
|
||||
@@ -0,0 +1,19 @@
|
||||
import sys, time
|
||||
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
|
||||
import box_walk as w
|
||||
name = sys.argv[1]; dirs = sys.argv[2:]
|
||||
EVF = open(f"{w.EV}/teardown.txt", "a", buffering=1)
|
||||
def say(*a):
|
||||
w.say(*a); EVF.write(" ".join(map(str, a)) + "\n")
|
||||
w.login()
|
||||
c, d = w.ctl("POST", f"/api/stacks/{name}/stop"); say(f"{name}: stop -> {c}")
|
||||
for _ in range(24):
|
||||
time.sleep(5)
|
||||
if w.stack(name).get("state") != "running": break
|
||||
c, d = w.ctl("POST", f"/api/stacks/{name}/remove", {"remove_hdd_data": False, "remove_backups": True}); say(f"{name}: remove (keep drive data, drop backups) -> {c} {str(d)[:120]}")
|
||||
time.sleep(5)
|
||||
for p in dirs:
|
||||
assert p.startswith("/mnt/felhom-drives/scratch_hdd/") and p.count("/") >= 5 and p.rstrip("/").split("/")[-1] == name, p
|
||||
say(f"{name}: tidy own folders by name:", w.guest("for p in " + " ".join(dirs) + "; do [ -d $p ] && rm -rf $p && echo removed $p; done; true").strip())
|
||||
st = w.stack(name)
|
||||
say(f"{name}: after: deployed={st.get('deployed')} leftovers={w.guest(f'ls -d /opt/docker/stacks/{name} 2>/dev/null; docker ps -a --filter label=com.docker.compose.project={name} --format {{{{.Names}}}}; docker volume ls -q --filter label=com.docker.compose.project={name}').strip()!r}")
|
||||
@@ -0,0 +1,12 @@
|
||||
#!/bin/sh
|
||||
# tun.sh <host> <visitor> <forged-left> <METHOD> <path> [body] [content-type] [extra-header]
|
||||
# ONE request through the SIMULATED tunnel (run in a curl container at 172.16.253.2, cloudflared's address).
|
||||
# Prints the HTTP status code only.
|
||||
H=$1; V=$2; F=$3; M=$4; P=$5; B=$6; CT=${7:-application/json}; X=$8
|
||||
if [ -n "$B" ]; then
|
||||
curl -sk -o /dev/null -w '%{http_code}' -X "$M" "https://traefik$P" -H "Host: $H" -H "X-Forwarded-For: $F, $V" \
|
||||
-H "CF-Connecting-IP: $V" -H "Content-Type: $CT" ${X:+-H "$X"} --data "$B"
|
||||
else
|
||||
curl -sk -o /dev/null -w '%{http_code}' -X "$M" "https://traefik$P" -H "Host: $H" -H "X-Forwarded-For: $F, $V" \
|
||||
-H "CF-Connecting-IP: $V" ${X:+-H "$X"}
|
||||
fi
|
||||
@@ -0,0 +1,40 @@
|
||||
import sys, time, json, secrets
|
||||
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
|
||||
import box_walk as w
|
||||
APP, SUB = "vikunja", "tasks"; H = f"{SUB}.{w.DOMAIN}"
|
||||
EVF = open(f"{w.EV}/vikunja.txt", "a", buffering=1)
|
||||
def say(*a):
|
||||
w.say(*a); EVF.write(" ".join(map(str, a)) + "\n")
|
||||
def tun(v, f, body):
|
||||
q = lambda s: "'" + s.replace("'", "'\\''") + "'"
|
||||
return w.guest(f"docker run --rm --network felhom-tunnel --ip 172.16.253.2 -v /root/tun.sh:/t.sh:ro curlimages/curl:8.11.1 "
|
||||
f"sh /t.sh {H} {v} {f} POST /api/v1/login {q(body)} application/json").strip().splitlines()[-1]
|
||||
pw = "Drill-" + secrets.token_hex(10)
|
||||
open(f"{w.SC}/.vkpw", "w").write(pw)
|
||||
def rounds(label):
|
||||
say(f"## {label} — {time.strftime('%FT%TZ', time.gmtime())}")
|
||||
bad = json.dumps({"username": "household", "password": "wrong"})
|
||||
r = [tun("198.51.100.66", f"10.5.0.{i}", bad) for i in range(1, 15)]
|
||||
say(" stranger 198.51.100.66, 14 wrong logins, a new forged leftmost each:", r)
|
||||
good = json.dumps({"username": "household", "password": pw})
|
||||
say(" household 203.0.113.10, the right password, at once:", tun("203.0.113.10", "203.0.113.10", good))
|
||||
w.login()
|
||||
say(f"##### vikunja on 9202 — controller {w.guest('docker inspect felhom-controller --format {{.Config.Image}}').strip()}")
|
||||
say("deploy ->", w.deploy(APP, SUB))
|
||||
w.wait_app(SUB, "/api/v1/info", tries=40)
|
||||
rc, code, body = w.app_curl(SUB, "/api/v1/register", "-H", "Content-Type: application/json", method="POST",
|
||||
data=json.dumps({"username": "household", "email": "household@example.com", "password": pw}))
|
||||
say("register the household's account (before the gate opens) ->", code)
|
||||
c, d = w.ctl("POST", f"/apps/{APP}/setup-gate/open", {}); say("setup-gate/open ->", c)
|
||||
time.sleep(20); w.wait_app(SUB, "/api/v1/info", tries=40)
|
||||
say("env:", w.guest("docker exec vikunja printenv VIKUNJA_SERVICE_IPEXTRACTIONMETHOD").strip())
|
||||
rounds("WITH the fix (VIKUNJA_SERVICE_IPEXTRACTIONMETHOD=xff)")
|
||||
D = "/opt/docker/stacks/vikunja"
|
||||
say("## CONTROL: the line removed from the stack's compose, restart through the product", w.guest(f"cp -p {D}/docker-compose.yml /root/vk.with; sed -i '/IPEXTRACTIONMETHOD/d' {D}/docker-compose.yml; grep -c IPEXTRACTION {D}/docker-compose.yml").strip())
|
||||
say(" waiting 70 s for the minute window"); time.sleep(70)
|
||||
c, d = w.ctl("POST", f"/api/stacks/{APP}/restart"); say(" restart ->", c); time.sleep(10); w.wait_app(SUB, "/api/v1/info", tries=40)
|
||||
say(" env:", w.guest("docker exec vikunja printenv VIKUNJA_SERVICE_IPEXTRACTIONMETHOD || echo '(unset)'").strip())
|
||||
rounds("CONTROL — the template before R-776 (direct)")
|
||||
say("## put the template's compose back:", w.guest(f"cp -p /root/vk.with {D}/docker-compose.yml; grep -c IPEXTRACTION {D}/docker-compose.yml").strip())
|
||||
c, d = w.ctl("POST", f"/api/stacks/{APP}/restart"); say(" restart ->", c); time.sleep(10)
|
||||
say(" env back:", w.guest("docker exec vikunja printenv VIKUNJA_SERVICE_IPEXTRACTIONMETHOD").strip())
|
||||
@@ -0,0 +1,36 @@
|
||||
import sys, time, json, secrets
|
||||
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
|
||||
import box_walk as w
|
||||
APP, SUB = "zipline", "img"; H = f"{SUB}.{w.DOMAIN}"
|
||||
EVF = open(f"{w.EV}/zipline.txt", "a", buffering=1)
|
||||
def say(*a):
|
||||
w.say(*a); EVF.write(" ".join(map(str, a)) + "\n")
|
||||
pw = "Drill-" + secrets.token_hex(10)
|
||||
w.guest(f"umask 077; printf '%s' '{json.dumps({'username': 'household', 'password': pw})}' > /root/.zlgood; chmod 644 /root/.zlgood")
|
||||
def burst(label):
|
||||
say(f"## {label} — {time.strftime('%FT%TZ', time.gmtime())}")
|
||||
bad = json.dumps({"username": "household", "password": "wrong"})
|
||||
say(" ", w.guest(f"docker run --rm --network felhom-tunnel --ip 172.16.253.2 -v /root/burst.sh:/b.sh:ro -v /root/.zlgood:/g:ro "
|
||||
f"curlimages/curl:8.11.1 sh /b.sh {H} /api/auth/login 10 '{bad}' /g").strip().splitlines()[-1])
|
||||
w.login()
|
||||
say(f"##### zipline on 9202 — controller {w.guest('docker inspect felhom-controller --format {{.Config.Image}}').strip()}")
|
||||
say("deploy ->", w.deploy(APP, SUB))
|
||||
w.wait_app(SUB, "/api/healthcheck", tries=60)
|
||||
rc, code, body = w.app_curl(SUB, "/api/setup", "-H", "Content-Type: application/json", method="POST",
|
||||
data=json.dumps({"username": "household", "password": pw}))
|
||||
say("first account through the front door (before the gate opens) ->", code, body[:80].replace(pw, "<pw>"))
|
||||
c, d = w.ctl("POST", f"/apps/{APP}/setup-gate/open", {}); say("setup-gate/open ->", c)
|
||||
time.sleep(15); w.wait_app(SUB, "/api/healthcheck", tries=40)
|
||||
env = lambda: [l for l in w.guest("docker inspect zipline --format '{{range .Config.Env}}{{println .}}{{end}}'").splitlines() if "TRUST" in l]
|
||||
say("env:", env())
|
||||
burst("WITH the fix (CORE_TRUST_PROXY=true + CORE_TRUSTED_PROXIES=172.16.0.0/12)")
|
||||
D = "/opt/docker/stacks/zipline"
|
||||
say("## CONTROL: both lines removed from the stack's compose", w.guest(f"cp -p {D}/docker-compose.yml /root/zl.with; sed -i '/CORE_TRUST_PROXY=/d;/CORE_TRUSTED_PROXIES=/d' {D}/docker-compose.yml; grep -c CORE_TRUST {D}/docker-compose.yml").strip())
|
||||
time.sleep(12)
|
||||
c, d = w.ctl("POST", f"/api/stacks/{APP}/restart"); say(" restart ->", c); time.sleep(10); w.wait_app(SUB, "/api/healthcheck", tries=40)
|
||||
say(" env:", env())
|
||||
burst("CONTROL — the template before R-776")
|
||||
say("## put the template's compose back:", w.guest(f"cp -p /root/zl.with {D}/docker-compose.yml; grep -c CORE_TRUST {D}/docker-compose.yml").strip())
|
||||
c, d = w.ctl("POST", f"/api/stacks/{APP}/restart"); say(" restart ->", c); time.sleep(10)
|
||||
say(" env back:", env())
|
||||
w.guest("rm -f /root/.zlgood")
|
||||
@@ -0,0 +1,20 @@
|
||||
##### vikunja on 9202 — controller gitea.dooplex.hu/admin/felhom-controller:0.299.0
|
||||
deploy -> True
|
||||
register the household's account (before the gate opens) -> 200
|
||||
setup-gate/open -> 200
|
||||
env: OCI runtime exec failed: exec failed: unable to start container process: exec: "printenv": executable file not found in $PATH
|
||||
## WITH the fix (VIKUNJA_SERVICE_IPEXTRACTIONMETHOD=xff) — 2026-10-06T06:30:16Z
|
||||
stranger 198.51.100.66, 14 wrong logins, a new forged leftmost each: ['403', '403', '403', '403', '403', '403', '403', '403', '403', '403', '429', '429', '429', '429']
|
||||
household 203.0.113.10, the right password, at once: 200
|
||||
## CONTROL: the line removed from the stack's compose, restart through the product 0
|
||||
waiting 70 s for the minute window
|
||||
restart -> 200
|
||||
env: OCI runtime exec failed: exec failed: unable to start container process: exec: "printenv": executable file not found in $PATH
|
||||
(unset)
|
||||
## CONTROL — the template before R-776 (direct) — 2026-10-06T06:32:36Z
|
||||
stranger 198.51.100.66, 14 wrong logins, a new forged leftmost each: ['403', '403', '403', '403', '403', '403', '403', '403', '403', '403', '429', '429', '429', '429']
|
||||
household 203.0.113.10, the right password, at once: 429
|
||||
## put the template's compose back: 1
|
||||
restart -> 200
|
||||
env back: OCI runtime exec failed: exec failed: unable to start container process: exec: "printenv": executable file not found in $PATH
|
||||
VIKUNJA_SERVICE_IPEXTRACTIONMETHOD=xff
|
||||
@@ -0,0 +1,15 @@
|
||||
##### zipline on 9202 — controller gitea.dooplex.hu/admin/felhom-controller:0.299.0
|
||||
deploy -> True
|
||||
first account through the front door (before the gate opens) -> 200 {"firstSetup":false,"user":{"id":"pun6ifh0vehx2edctxo1w55g","username":"househol
|
||||
setup-gate/open -> 200
|
||||
env: ['CORE_TRUSTED_PROXIES=172.16.0.0/12', 'CORE_TRUST_PROXY=true']
|
||||
## WITH the fix (CORE_TRUST_PROXY=true + CORE_TRUSTED_PROXIES=172.16.0.0/12) — 2026-10-06T06:36:47Z
|
||||
stranger: 400 400 400 400 400 400 400 429 429 429 | household: 200
|
||||
## CONTROL: both lines removed from the stack's compose 0
|
||||
restart -> 200
|
||||
env: []
|
||||
## CONTROL — the template before R-776 — 2026-10-06T06:37:31Z
|
||||
stranger: 400 400 400 400 400 400 400 429 429 429 | household: 429
|
||||
## put the template's compose back: 2
|
||||
restart -> 200
|
||||
env back: ['CORE_TRUST_PROXY=true', 'CORE_TRUSTED_PROXIES=172.16.0.0/12']
|
||||
@@ -0,0 +1,8 @@
|
||||
== R-889 read-back 2026-10-06T06:46:57Z: hub customer page (from the box's report) vs df in the guest (a different channel)
|
||||
-- demo-hp hub: SSD 23% 14.7 / 68.7 GB NVME 1TB 6% 54.6 / 937.8 GB
|
||||
df: /mnt/sys_drive=23% /mnt/felhom-drives/hdd_1=7%
|
||||
-- demo-felhom hub: SSD 1% 1.6 / 245.0 GB
|
||||
df: /mnt/sys_drive=1%
|
||||
Storage SSD 6% 4.1 / 68.7 GB Adatlemez 0% 0.2 / 48.9 GB Bac
|
||||
Controller version 0.299.0
|
||||
Controller elindult (0.299.0)
|
||||
@@ -0,0 +1,3 @@
|
||||
--- FAIL: TestR889_EveryDiskPercentIsTheDFOne (0.05s)
|
||||
dfpercent_test.go:66: a disk percent divides by the whole filesystem (not df's used/(used+avail)) — use DFUsedPercent:
|
||||
../../internal/system/mounts_linux.go:85: info.UsedPercent = float64(used) / float64(total) * 100
|
||||
@@ -26,6 +26,18 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-06 (morning) — R-889 delivered, the held catalog fixes proven and pushed
|
||||
|
||||
The full text of every row below: `git show e6cfe6d6:documentation/backlog/OPEN-ITEMS.md`.
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|---|---|---|---|
|
||||
| **R-613** | **[P2-MEDIUM] `uptime-kuma` parks on its setup wizard with no login and no monitors, and the box tells the household it is HEALTHY.** (P3) | CLOSED 2026-10-06 — FIXED, PROVEN ON 9202 AND PUSHED: nextcloud's probe needs a finished install | catalog `c5674ce` (live `3896cb0`): `body_contains: '"installed":true'`. Measured on 9202: `status.php` = `{"installed":true,"maintenance":false,…}` and the controller reads the app `running` with the new probe (`live-9202/nextcloud.txt`). The sweep found no other wizard that reads healthy unnoticed. |
|
||||
| **R-621** | **[P2-MEDIUM] A held update DESTROYS the evidence of why it failed: `failAndHold` runs `compose down`, the failing containers are removed, and their output is gone before anyone — household, operator or the next session — can read it.** (P4) | CLOSED 2026-10-06 — FIXED AND DELIVERED: a held update keeps the app's log and shows it | controller `0b93e1a` (v0.298.0: logs page + API) + `0f2eab7` (v0.299.0: the hold panel links to it, `09` decision 136). Delivered 2026-10-06 06:17Z: controller v0.299.0 (golden 0.299.0, floors for demo-hp, demo-felhom, tester-1; `audits/morning-after-2026-10-06/delivery/`). |
|
||||
| **R-889** | **The disk percent the box shows and alarms on is `used / total`, not what `df` shows (`used / (used + available)`), so on a full system disk it reads ~5 points LOW — and the fill alarms (85 % / 95 %) fire that much late.** (P3) | CLOSED 2026-10-06 — FIXED AND DELIVERED (operator ruling, `09` §3 decision 138): every disk percent is df's | controller `bee2c2d` (v0.299.0): `system.DFUsedPercent` (used / (used + available)) serves the four places that computed one; tests on numbers measured on demo-hp 9201 + a source scan, red-proved (`audits/morning-after-2026-10-06/r889-red-proof.txt`). Cause MEASURED: the 5 % reserved for root was in the denominator (demo-hp `/mnt/sys_drive`: `stat -f` blocks 18016108, free 14150899, avail 13229302 → old 21.5 %, df 23 %). **Read back after delivery (hub page vs `df` in the guest):** demo-hp SSD 21 % → **23 %** (df 23 %), NVMe 6 % (df 7 %, df rounds up), demo-felhom SSD 1 % (df 1 %) (`r889-readback.txt`). The tile now says „Rendszer” (it measured the docker data volume, not /). Alarm levels unchanged — a full disk alarms a little earlier. Delivered 2026-10-06 06:17Z: controller v0.299.0 (golden 0.299.0, floors for demo-hp, demo-felhom, tester-1; `audits/morning-after-2026-10-06/delivery/`). |
|
||||
| **R-776** | **[P3-LOW] Right-walking catalog apps need ONE setting to see each visitor since v0.286 (R-753); without it they keep the tunnel's one address (a stranger can still trip their per-address limits for everyone).** (P3) | CLOSED 2026-10-06 — FIXED, PROVEN ON 9202 AND PUSHED: four apps see each visitor behind the tunnel | catalog `462e66f` kimai, `ccc2be5` zipline, `5f7bf77` vikunja, `1d3af8e` nextcloud (live catalog `3896cb0`, 06:46Z). Measured on 9202 through the simulated tunnel, each with a control (the line removed, restart through the product): vikunja household 200 vs 429; zipline 200 vs 429; kimai ok vs THROTTLED; nextcloud attempts counted for the visitor 6 vs 0 — and a stranger changing the forged leftmost address was throttled every time (`audits/morning-after-2026-10-06/live-9202/`). |
|
||||
| **R-585** | **[P3-LOW] Six event producers still send Hungarian only, so an English household can see one Hungarian line in some mails.** (P3) | CLOSED 2026-10-06 — FIXED AND DELIVERED: every customer-facing producer follows the household's language | controller `ca89e70` (v0.298.0) + `ff1758a` (v0.299.0); `local_api_endpoint_drift` is operator-only and English by design (the row's list was wrong about it). Delivered 2026-10-06 06:17Z: controller v0.299.0 (golden 0.299.0, floors for demo-hp, demo-felhom, tester-1; `audits/morning-after-2026-10-06/delivery/`). |
|
||||
|
||||
## 2026-10-06 (morning) — installer 1.32.0 published
|
||||
|
||||
The full text of every row below: `git show 32a15208:documentation/backlog/OPEN-ITEMS.md`.
|
||||
|
||||
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user