admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Campaign 10: RESOLVED — the backup_target_* silence was transient and self-recovered; SQLITE_BUSY drops are absorbed by retry

Browse Source 
Both halves of the disposition were run and neither survived as a finding.

The I1/I1-pair violations cluster at cycles 31-33 and nowhere else across 39
cycles; c34-c39 are clean, so it recovered with no intervention. Final tally I1
37 PASS / 2 VIOLATION, I1-pair 36 PASS / 3 VIOLATION. On the quiesced box one slow
detach with 4 minutes either side produced a perfect pair. And the alarming
false-healthy (mentes bound=False while degraded=false) does not survive
quiescence - I had been reading the two halves at different instants of a detach.
No R-n.

Separately cleared: the hub's SQLITE_BUSY event drops. 7 in 24h including one for
the real customer demo-felhom, and the hub does return 500 with notification
dispatch only after a successful save - so a lost event would be a lost alarm. But
the controller retries 3 times and ZERO events exhausted their attempts; the
07:04:39 drop landed at 07:04:42. Nothing lost. Only cosmetic note: the ERROR line
reads like data loss and is not.
This commit is contained in:
authentik Default Admin
2026-08-02 07:10:49 +02:00
parent 1931dfcb0c
commit 405a795e32
6 changed files with 9018 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/tests/campaign10-evidence-2026-07-31/OPEN-backup-target-pair-went-silent.md → documentation/tests/campaign10-evidence-2026-07-31/RESOLVED-backup-target-pair-went-silent.md
+69
View File
@@ -62,3 +62,72 @@ After run 2b ends, let the box **quiesce with both drives attached**, then:
after its first firing is worse than no alarm.
Only then does it earn an `R-n` or a retraction.
---
# RESOLVED (2026-08-02) — transient, self-recovered. NOT filed.
Both halves of the disposition were run. Neither survived.
## 1. It self-recovered, mid-run, with no intervention
The violations are **clustered at cycles 31-33 and nowhere else** across 39 cycles:
c28 I1 PASS c28 I1-pair PASS
c29 I1 PASS c29 I1-pair PASS
c30 I1 PASS c30 I1-pair PASS
c31 I1 PASS c31 I1-pair VIOLATION
c32 I1 VIOLATION c32 I1-pair VIOLATION
c33 I1 VIOLATION c33 I1-pair VIOLATION
c34 I1 PASS c34 I1-pair PASS <-- recovered on its own
c35..c39 all PASS
Final tally **I1 37 PASS / 2 VIOLATION, I1-pair 36 PASS / 3 VIOLATION** — a ~3-cycle window inside a
13.5-hour run of sustained 2-minute detach/reattach cycling.
## 2. A slow cycle on the quiesced box works perfectly
With the run finished and the box quiet, one detach with 4 minutes either side:
07:00:08 backup_target_absent (error) — A rendszermentés meghajtója nem érhető el: Mentes
07:04:42 backup_target_restored (info) — A rendszermentés meghajtója újra elérhető: Mentes
07:06:37 health_recovered (info) — Rendszer állapot helyreállt: ok (volt: warn)
## 3. The "false healthy" was a mid-flux sample, not a stuck state
The alarming pair (`mentes bound_under_parent=False` **and** `degraded: false`) does not survive
quiescence. Three minutes after the run ended:
drives: adatok bound=True mentes bound=True backup_target=True
target: {"degraded": false, "known": true, "label": "Mentes", "target": "felhom-backup"}
Consistent. I was sampling a system mid-detach and reading the two halves at different instants.
## Verdict
**No `R-n`.** The target alarm does not fire during a narrow window under detach/reattach cycling far
faster than any real box sees, and it resumes by itself. Recorded so a future session that sees the
same three-cycle cluster knows it was chased to the ground and what settled it.
---
# Also investigated and cleared: the hub's SQLITE_BUSY event drops
Chased during the same test and **also not a finding**, recorded because the log line is alarming:
[ERROR] Failed to save event from c10-soak: database is locked (5) (SQLITE_BUSY)
7 occurrences in 24 h, **including one for the real customer `demo-felhom`** — so it is not a
campaign artifact. The hub returns HTTP 500 on the failed save (`handler.go:1715`) and dispatches
notifications only *after* a successful save, so a genuinely lost event would be a lost alarm.
**But the controller retries 3 times**, and **0 events exhausted their attempts** across the whole
run (`grep -c "Event push failed after 3 attempts"` = 0). The 07:04:39 drop landed 3 seconds later:
07:04:39 [ERROR] Failed to save event … SQLITE_BUSY
07:04:42 [INFO] Event from c10-soak: backup_target_restored (info) … <-- same event, retried
**Nothing was lost.** The one observation worth keeping is cosmetic: `[ERROR] Failed to save event`
reads like data loss and is not — a reader would reasonably conclude an alarm had been dropped. The
same contention also skipped monitor ticks (Host OOB, Offsite, restore-test, log-bundle purge), which
are periodic and simply run again.
documentation/tests/campaign10-evidence-2026-07-31/state/growth.tsv
+7936
View File
File diff suppressed because it is too large Load Diff
documentation/tests/campaign10-evidence-2026-07-31/state/journal.tsv
+875
View File
@@ -1 +1,876 @@
cycle utc invariant verdict detail
1 2026-08-01T17:53:31Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
1 2026-08-01T17:53:34Z I3 PASS adatok bound_under_parent=False while absent
1 2026-08-01T17:55:47Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
1 2026-08-01T17:55:49Z I4 PASS guest init boot before=1785606256 after=1785606256 (must be equal)
1 2026-08-01T17:57:25Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.349727489s", "last_run": "2026-08-01T17:57:13.524751498Z", "success": true}, "enabled": true, "running": false}
1 2026-08-01T17:57:25Z KILL-AGENT PASS agent active after kill=True; subsequent backup ok=True
1 2026-08-01T17:58:07Z BACKUP PASS {"db_dump": {"count": 1, "duration": "34.28564693s", "last_run": "2026-08-01T17:58:01.538710099Z", "success": true}, "enabled": true, "running": false}
1 2026-08-01T17:59:24Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.309445496s", "last_run": "2026-08-01T17:59:11.578099671Z", "success": true}, "enabled": true, "running": false}
1 2026-08-01T18:00:35Z I7 PASS app=rallly want=C10-C001-A-195822 got=C10-C001-A-195822 snap=helyi restore_ok=True rto=41.9s
1 2026-08-01T18:02:51Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
1 2026-08-01T18:02:54Z I3 PASS mentes bound_under_parent=False while absent
1 2026-08-01T18:05:06Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
1 2026-08-01T18:05:09Z I4 PASS guest init boot before=1785606256 after=1785606256 (must be equal)
1 2026-08-01T18:05:53Z REDEPLOY PASS grafana redeployed
1 2026-08-01T18:06:43Z KILL-CTRL PASS controller recovered after kill
1 2026-08-01T18:06:46Z I5 PASS degraded=False absent_events=0 target=felhom-backup
1 2026-08-01T18:06:46Z I6 PASS healthy state must carry no degraded message; got=None
1 2026-08-01T18:06:48Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
1 2026-08-01T18:06:48Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
1 2026-08-01T18:06:50Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
1 2026-08-01T18:06:52Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
1 2026-08-01T18:06:54Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
1 2026-08-01T18:06:56Z I11 PASS leaked=[]
2 2026-08-01T18:07:42Z REDEPLOY PASS grafana redeployed
2 2026-08-01T18:09:58Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
2 2026-08-01T18:10:00Z I3 PASS adatok bound_under_parent=False while absent
2 2026-08-01T18:12:13Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
2 2026-08-01T18:12:16Z I4 PASS guest init boot before=1785606256 after=1785606256 (must be equal)
2 2026-08-01T18:13:06Z KILL-CTRL PASS controller recovered after kill
2 2026-08-01T18:13:36Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.000242906s", "last_run": "2026-08-01T18:13:35.28902448Z", "success": true}, "enabled": true, "running": false}
2 2026-08-01T18:17:24Z HARD-RESET PASS VM returned=True apps_ready=True canaries_intact=True (a hard reset mid-write must not corrupt them)
2 2026-08-01T18:18:27Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.694681688s", "last_run": "2026-08-01T18:18:16.120290791Z", "success": true}, "enabled": true, "running": false}
2 2026-08-01T18:19:37Z I7 PASS app=rallly want=C10-C002-A-201726 got=C10-C002-A-201726 snap=helyi restore_ok=True rto=41.6s
2 2026-08-01T18:21:52Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
2 2026-08-01T18:21:55Z I3 PASS mentes bound_under_parent=False while absent
2 2026-08-01T18:24:08Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
2 2026-08-01T18:24:11Z I4 PASS guest init boot before=1785608199 after=1785608199 (must be equal)
2 2026-08-01T18:24:13Z I5 PASS degraded=False absent_events=0 target=felhom-backup
2 2026-08-01T18:24:13Z I6 PASS healthy state must carry no degraded message; got=None
2 2026-08-01T18:24:15Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
2 2026-08-01T18:24:15Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
2 2026-08-01T18:24:17Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
2 2026-08-01T18:24:19Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
2 2026-08-01T18:24:21Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
2 2026-08-01T18:24:24Z I11 PASS leaked=[]
3 2026-08-01T18:24:55Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.099984037s", "last_run": "2026-08-01T18:24:55.529304167Z", "success": true}, "enabled": true, "running": false}
3 2026-08-01T18:25:39Z REDEPLOY PASS homebox redeployed
3 2026-08-01T18:27:55Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
3 2026-08-01T18:27:58Z I3 PASS mentes bound_under_parent=False while absent
3 2026-08-01T18:30:11Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
3 2026-08-01T18:30:13Z I4 PASS guest init boot before=1785608199 after=1785608199 (must be equal)
3 2026-08-01T18:31:29Z REBOOT-VM PASS controller back after full VM reboot
3 2026-08-01T18:33:45Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
3 2026-08-01T18:33:48Z I3 PASS adatok bound_under_parent=False while absent
3 2026-08-01T18:36:01Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
3 2026-08-01T18:36:03Z I4 PASS guest init boot before=1785609078 after=1785609078 (must be equal)
3 2026-08-01T18:36:53Z KILL-CTRL PASS controller recovered after kill
3 2026-08-01T18:37:49Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.561742447s", "last_run": "2026-08-01T18:37:49.314975637Z", "success": true}, "enabled": true, "running": false}
3 2026-08-01T18:39:14Z I7 PASS app=rallly want=C10-C003-A-203656 got=C10-C003-A-203656 snap=helyi restore_ok=True rto=45.0s
3 2026-08-01T18:39:17Z I5 PASS degraded=False absent_events=0 target=felhom-backup
3 2026-08-01T18:39:17Z I6 PASS healthy state must carry no degraded message; got=None
3 2026-08-01T18:39:19Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
3 2026-08-01T18:39:19Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
3 2026-08-01T18:39:21Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
3 2026-08-01T18:39:23Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
3 2026-08-01T18:39:25Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
3 2026-08-01T18:39:28Z I11 PASS leaked=[]
4 2026-08-01T18:41:46Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
4 2026-08-01T18:41:49Z I3 PASS adatok bound_under_parent=False while absent
4 2026-08-01T18:44:02Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
4 2026-08-01T18:44:05Z I4 PASS guest init boot before=1785609078 after=1785609078 (must be equal)
4 2026-08-01T18:44:49Z REDEPLOY PASS grafana redeployed
4 2026-08-01T18:45:20Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.432027836s", "last_run": "2026-08-01T18:45:19.344957434Z", "success": true}, "enabled": true, "running": false}
4 2026-08-01T18:46:00Z REBOOT PASS controller back after guest reboot
4 2026-08-01T18:47:18Z BACKUP PASS {"db_dump": {"count": 1, "duration": "35.597712303s", "last_run": "2026-08-01T18:47:13.540222182Z", "success": true}, "enabled": true, "running": false}
4 2026-08-01T18:48:31Z I7 PASS app=rallly want=C10-C004-A-204616 got=C10-C004-A-204616 snap=helyi restore_ok=True rto=42.1s
4 2026-08-01T18:50:46Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
4 2026-08-01T18:50:49Z I3 PASS mentes bound_under_parent=False while absent
4 2026-08-01T18:53:02Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
4 2026-08-01T18:53:04Z I4 PASS guest init boot before=1785609940 after=1785609940 (must be equal)
4 2026-08-01T18:53:53Z KILL-CTRL PASS controller recovered after kill
4 2026-08-01T18:53:56Z I5 PASS degraded=False absent_events=0 target=felhom-backup
4 2026-08-01T18:53:56Z I6 PASS healthy state must carry no degraded message; got=None
4 2026-08-01T18:53:58Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
4 2026-08-01T18:53:58Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
4 2026-08-01T18:54:00Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
4 2026-08-01T18:54:02Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
4 2026-08-01T18:54:04Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
4 2026-08-01T18:54:07Z I11 PASS leaked=[]
5 2026-08-01T18:56:26Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
5 2026-08-01T18:56:29Z I3 PASS adatok bound_under_parent=False while absent
5 2026-08-01T18:58:42Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
5 2026-08-01T18:58:45Z I4 PASS guest init boot before=1785609940 after=1785609940 (must be equal)
5 2026-08-01T18:59:15Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.578451797s", "last_run": "2026-08-01T18:59:14.920949079Z", "success": true}, "enabled": true, "running": false}
5 2026-08-01T18:59:56Z CONCURRENCY PASS backup+restore raced; restore-status op=None running=False (recorded, not asserted)
5 2026-08-01T19:00:41Z REDEPLOY PASS homebox redeployed
5 2026-08-01T19:01:33Z KILL-CTRL PASS controller recovered after kill
5 2026-08-01T19:02:30Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.651988711s", "last_run": "2026-08-01T19:02:28.226002166Z", "success": true}, "enabled": true, "running": false}
5 2026-08-01T19:03:46Z I7 PASS app=rallly want=C10-C005-A-210135 got=C10-C005-A-210135 snap=helyi restore_ok=True rto=42.1s
5 2026-08-01T19:06:02Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
5 2026-08-01T19:06:05Z I3 PASS mentes bound_under_parent=False while absent
5 2026-08-01T19:08:20Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
5 2026-08-01T19:08:26Z I4 PASS guest init boot before=1785609940 after=1785609940 (must be equal)
5 2026-08-01T19:08:32Z I5 PASS degraded=False absent_events=0 target=felhom-backup
5 2026-08-01T19:08:32Z I6 PASS healthy state must carry no degraded message; got=None
5 2026-08-01T19:08:34Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
5 2026-08-01T19:08:34Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
5 2026-08-01T19:08:37Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
5 2026-08-01T19:08:40Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
5 2026-08-01T19:08:43Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
5 2026-08-01T19:08:49Z I11 PASS leaked=[]
6 2026-08-01T19:10:15Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.113626659s", "last_run": "2026-08-01T19:10:03.742893809Z", "success": true}, "enabled": true, "running": false}
6 2026-08-01T19:11:26Z I7 PASS app=rallly want=C10-C006-A-210908 got=C10-C006-A-210908 snap=helyi restore_ok=True rto=41.6s
6 2026-08-01T19:13:42Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
6 2026-08-01T19:13:45Z I3 PASS mentes bound_under_parent=False while absent
6 2026-08-01T19:15:58Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
6 2026-08-01T19:16:00Z I4 PASS guest init boot before=1785609940 after=1785609940 (must be equal)
6 2026-08-01T19:16:42Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.189811651s", "last_run": "2026-08-01T19:16:30.171776674Z", "success": true}, "enabled": true, "running": false}
6 2026-08-01T19:19:03Z I1-under-load PASS target pulled DURING a backup -> got=backup_target_absent (want backup_target_absent)
6 2026-08-01T19:21:21Z I1-under-load-recover PASS after return degraded=False target=felhom-backup
6 2026-08-01T19:22:11Z KILL-CTRL PASS controller recovered after kill
6 2026-08-01T19:22:54Z REDEPLOY PASS grafana redeployed
6 2026-08-01T19:25:10Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
6 2026-08-01T19:25:13Z I3 PASS adatok bound_under_parent=False while absent
6 2026-08-01T19:27:26Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
6 2026-08-01T19:27:28Z I4 PASS guest init boot before=1785609940 after=1785609940 (must be equal)
6 2026-08-01T19:27:30Z I5 PASS degraded=False absent_events=0 target=felhom-backup
6 2026-08-01T19:27:30Z I6 PASS healthy state must carry no degraded message; got=None
6 2026-08-01T19:27:32Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
6 2026-08-01T19:27:32Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
6 2026-08-01T19:27:34Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
6 2026-08-01T19:27:36Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
6 2026-08-01T19:27:38Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
6 2026-08-01T19:27:40Z I11 PASS leaked=[]
7 2026-08-01T19:28:27Z REDEPLOY PASS homebox redeployed
7 2026-08-01T19:29:16Z KILL-CTRL PASS controller recovered after kill
7 2026-08-01T19:30:08Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.132405112s", "last_run": "2026-08-01T19:30:07.380517766Z", "success": true}, "enabled": true, "running": false}
7 2026-08-01T19:31:20Z I7 PASS app=rallly want=C10-C007-A-212918 got=C10-C007-A-212918 snap=helyi restore_ok=True rto=41.4s
7 2026-08-01T19:31:56Z FILL-DRIVE PASS near-full (G 300M 100% /mnt/mentes) backup outcome recorded: {"db_dump": {"count": 1, "duration": "26.935248242s", "last_run": "2026-08-01T19:31:54.373462186Z", "success": true}, "enabled": true, "running": false}
7 2026-08-01T19:34:12Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
7 2026-08-01T19:34:15Z I3 PASS mentes bound_under_parent=False while absent
7 2026-08-01T19:36:28Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
7 2026-08-01T19:36:31Z I4 PASS guest init boot before=1785609940 after=1785609940 (must be equal)
7 2026-08-01T19:38:47Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
7 2026-08-01T19:38:50Z I3 PASS adatok bound_under_parent=False while absent
7 2026-08-01T19:41:03Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
7 2026-08-01T19:41:05Z I4 PASS guest init boot before=1785609940 after=1785609940 (must be equal)
7 2026-08-01T19:41:34Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.147600604s", "last_run": "2026-08-01T19:41:34.551664402Z", "success": true}, "enabled": true, "running": false}
7 2026-08-01T19:41:37Z I5 PASS degraded=False absent_events=0 target=felhom-backup
7 2026-08-01T19:41:37Z I6 PASS healthy state must carry no degraded message; got=None
7 2026-08-01T19:41:39Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
7 2026-08-01T19:41:39Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
7 2026-08-01T19:41:41Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
7 2026-08-01T19:41:43Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
7 2026-08-01T19:41:46Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
7 2026-08-01T19:41:48Z I11 PASS leaked=[]
8 2026-08-01T19:42:35Z REDEPLOY PASS grafana redeployed
8 2026-08-01T19:43:23Z KILL-CTRL PASS controller recovered after kill
8 2026-08-01T19:44:14Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.843783827s", "last_run": "2026-08-01T19:44:14.091033776Z", "success": true}, "enabled": true, "running": false}
8 2026-08-01T19:45:27Z I7 PASS app=rallly want=C10-C008-A-214325 got=C10-C008-A-214325 snap=helyi restore_ok=True rto=41.4s
8 2026-08-01T19:45:56Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.117775321s", "last_run": "2026-08-01T19:45:56.507733485Z", "success": true}, "enabled": true, "running": false}
8 2026-08-01T19:48:12Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
8 2026-08-01T19:48:15Z I3 PASS mentes bound_under_parent=False while absent
8 2026-08-01T19:50:28Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
8 2026-08-01T19:50:31Z I4 PASS guest init boot before=1785609940 after=1785609940 (must be equal)
8 2026-08-01T19:53:07Z I4-abort PASS aborted-in-place: bound=False (want False), device still present=True, opts= rw,relatime,abort,emergency_ro 0 0
8 2026-08-01T19:53:11Z I3-abort PASS calibre-web must be stopped while its namespace is aborted; running=False
8 2026-08-01T19:56:01Z I4-abort-recover PASS after re-seat: bound=True app_up=True
8 2026-08-01T19:58:17Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
8 2026-08-01T19:58:21Z I3 PASS adatok bound_under_parent=False while absent
8 2026-08-01T20:00:34Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
8 2026-08-01T20:00:36Z I4 PASS guest init boot before=1785609940 after=1785609940 (must be equal)
8 2026-08-01T20:00:39Z I5 PASS degraded=False absent_events=0 target=felhom-backup
8 2026-08-01T20:00:39Z I6 PASS healthy state must carry no degraded message; got=None
8 2026-08-01T20:00:41Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
8 2026-08-01T20:00:41Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
8 2026-08-01T20:00:43Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
8 2026-08-01T20:00:45Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
8 2026-08-01T20:00:47Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
8 2026-08-01T20:00:49Z I11 PASS leaked=[]
9 2026-08-01T20:01:35Z REDEPLOY PASS homebox redeployed
9 2026-08-01T20:02:24Z KILL-CTRL PASS controller recovered after kill
9 2026-08-01T20:03:50Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.196358521s", "last_run": "2026-08-01T20:03:49.840369464Z", "success": true}, "enabled": true, "running": false}
9 2026-08-01T20:03:50Z KILL-AGENT PASS agent active after kill=True; subsequent backup ok=True
9 2026-08-01T20:06:05Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
9 2026-08-01T20:06:08Z I3 PASS adatok bound_under_parent=False while absent
9 2026-08-01T20:08:21Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
9 2026-08-01T20:08:24Z I4 PASS guest init boot before=1785609940 after=1785609940 (must be equal)
9 2026-08-01T20:10:40Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
9 2026-08-01T20:10:43Z I3 PASS mentes bound_under_parent=False while absent
9 2026-08-01T20:12:56Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
9 2026-08-01T20:12:58Z I4 PASS guest init boot before=1785609940 after=1785609940 (must be equal)
9 2026-08-01T20:14:01Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.378159531s", "last_run": "2026-08-01T20:13:48.95740933Z", "success": true}, "enabled": true, "running": false}
9 2026-08-01T20:15:13Z I7 PASS app=rallly want=C10-C009-A-221300 got=C10-C009-A-221300 snap=helyi restore_ok=True rto=41.9s
9 2026-08-01T20:15:42Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.074679758s", "last_run": "2026-08-01T20:15:42.24037136Z", "success": true}, "enabled": true, "running": false}
9 2026-08-01T20:15:45Z I5 PASS degraded=False absent_events=0 target=felhom-backup
9 2026-08-01T20:15:45Z I6 PASS healthy state must carry no degraded message; got=None
9 2026-08-01T20:15:47Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
9 2026-08-01T20:15:47Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
9 2026-08-01T20:15:49Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
9 2026-08-01T20:15:51Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
9 2026-08-01T20:15:53Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
9 2026-08-01T20:15:56Z I11 PASS leaked=[]
10 2026-08-01T20:18:13Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
10 2026-08-01T20:18:16Z I3 PASS mentes bound_under_parent=False while absent
10 2026-08-01T20:20:29Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
10 2026-08-01T20:20:32Z I4 PASS guest init boot before=1785609940 after=1785609940 (must be equal)
10 2026-08-01T20:22:48Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
10 2026-08-01T20:22:51Z I3 PASS adatok bound_under_parent=False while absent
10 2026-08-01T20:25:04Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
10 2026-08-01T20:25:06Z I4 PASS guest init boot before=1785609940 after=1785609940 (must be equal)
10 2026-08-01T20:38:40Z HARD-RESET VIOLATION VM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
10 2026-08-01T20:39:25Z REDEPLOY PASS grafana redeployed
10 2026-08-01T20:49:32Z I7-SKIP PASS apps not ready before seed — atom skipped, NOT counted as a violation
10 2026-08-01T20:50:02Z BACKUP PASS {"db_dump": {"count": 1, "duration": "18.75452604s", "last_run": "2026-08-01T20:49:54.258246299Z", "success": true}, "enabled": true, "running": false}
10 2026-08-01T20:50:51Z KILL-CTRL PASS controller recovered after kill
10 2026-08-01T20:50:54Z I5 PASS degraded=False absent_events=0 target=felhom-backup
10 2026-08-01T20:50:54Z I6 PASS healthy state must carry no degraded message; got=None
10 2026-08-01T20:50:56Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
10 2026-08-01T20:50:56Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
10 2026-08-01T20:50:58Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
10 2026-08-01T20:51:00Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
10 2026-08-01T20:51:02Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
10 2026-08-01T20:51:05Z I11 PASS leaked=[]
11 2026-08-01T20:51:57Z KILL-CTRL PASS controller recovered after kill
11 2026-08-01T20:53:13Z REBOOT-VM PASS controller back after full VM reboot
11 2026-08-01T20:53:43Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.292223355s", "last_run": "2026-08-01T20:53:42.948096956Z", "success": true}, "enabled": true, "running": false}
11 2026-08-01T20:55:59Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
11 2026-08-01T20:56:02Z I3 PASS mentes bound_under_parent=False while absent
11 2026-08-01T20:58:15Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
11 2026-08-01T20:58:25Z I4 PASS guest init boot before=1785617581 after=1785617581 (must be equal)
11 2026-08-01T21:00:50Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
11 2026-08-01T21:00:56Z I3 PASS adatok bound_under_parent=False while absent
11 2026-08-01T21:03:09Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
11 2026-08-01T21:03:12Z I4 PASS guest init boot before=1785617581 after=1785617581 (must be equal)
11 2026-08-01T21:04:44Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.289422332s", "last_run": "2026-08-01T21:04:39.072258434Z", "success": true}, "enabled": true, "running": false}
11 2026-08-01T21:06:54Z I7 PASS app=rallly want=C10-C011-A-230315 got=C10-C011-A-230315 snap=helyi restore_ok=True rto=49.6s
11 2026-08-01T21:08:02Z REDEPLOY PASS homebox redeployed
11 2026-08-01T21:08:06Z I5 PASS degraded=False absent_events=0 target=felhom-backup
11 2026-08-01T21:08:06Z I6 PASS healthy state must carry no degraded message; got=None
11 2026-08-01T21:08:14Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
11 2026-08-01T21:08:14Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
11 2026-08-01T21:08:20Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
11 2026-08-01T21:08:24Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
11 2026-08-01T21:08:27Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
11 2026-08-01T21:08:30Z I11 PASS leaked=[]
12 2026-08-01T21:09:14Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.545961681s", "last_run": "2026-08-01T21:09:04.966067493Z", "success": true}, "enabled": true, "running": false}
12 2026-08-01T21:10:07Z REDEPLOY PASS homebox redeployed
12 2026-08-01T21:12:29Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
12 2026-08-01T21:12:32Z I3 PASS adatok bound_under_parent=False while absent
12 2026-08-01T21:14:45Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
12 2026-08-01T21:14:48Z I4 PASS guest init boot before=1785617581 after=1785617581 (must be equal)
12 2026-08-01T21:15:37Z KILL-CTRL PASS controller recovered after kill
12 2026-08-01T21:17:53Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
12 2026-08-01T21:17:56Z I3 PASS mentes bound_under_parent=False while absent
12 2026-08-01T21:20:10Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
12 2026-08-01T21:20:12Z I4 PASS guest init boot before=1785617581 after=1785617581 (must be equal)
12 2026-08-01T21:21:07Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.670047182s", "last_run": "2026-08-01T21:21:07.162430191Z", "success": true}, "enabled": true, "running": false}
12 2026-08-01T21:22:18Z I7 PASS app=rallly want=C10-C012-A-232015 got=C10-C012-A-232015 snap=helyi restore_ok=True rto=41.4s
12 2026-08-01T21:22:58Z REBOOT PASS controller back after guest reboot
12 2026-08-01T21:23:01Z I5 PASS degraded=False absent_events=0 target=felhom-backup
12 2026-08-01T21:23:01Z I6 PASS healthy state must carry no degraded message; got=None
12 2026-08-01T21:23:03Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
12 2026-08-01T21:23:03Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
12 2026-08-01T21:23:05Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
12 2026-08-01T21:23:07Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
12 2026-08-01T21:23:09Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
12 2026-08-01T21:23:11Z I11 PASS leaked=[]
13 2026-08-01T21:23:57Z REDEPLOY PASS grafana redeployed
13 2026-08-01T21:24:39Z BACKUP PASS {"db_dump": {"count": 1, "duration": "29.007233593s", "last_run": "2026-08-01T21:24:28.362712575Z", "success": true}, "enabled": true, "running": false}
13 2026-08-01T21:25:28Z KILL-CTRL PASS controller recovered after kill
13 2026-08-01T21:27:44Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
13 2026-08-01T21:27:47Z I3 PASS mentes bound_under_parent=False while absent
13 2026-08-01T21:30:00Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
13 2026-08-01T21:30:02Z I4 PASS guest init boot before=1785619357 after=1785619357 (must be equal)
13 2026-08-01T21:30:41Z CONCURRENCY PASS backup+restore raced; restore-status op=None running=False (recorded, not asserted)
13 2026-08-01T21:31:44Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.58110407s", "last_run": "2026-08-01T21:31:33.003741001Z", "success": true}, "enabled": true, "running": false}
13 2026-08-01T21:32:43Z I7 PASS app=rallly want=C10-C013-A-233043 got=C10-C013-A-233043 snap=helyi restore_ok=True rto=29.1s
13 2026-08-01T21:34:59Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
13 2026-08-01T21:35:02Z I3 PASS adatok bound_under_parent=False while absent
13 2026-08-01T21:37:15Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
13 2026-08-01T21:37:17Z I4 PASS guest init boot before=1785619357 after=1785619357 (must be equal)
13 2026-08-01T21:37:20Z I5 PASS degraded=False absent_events=0 target=felhom-backup
13 2026-08-01T21:37:20Z I6 PASS healthy state must carry no degraded message; got=None
13 2026-08-01T21:37:22Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
13 2026-08-01T21:37:22Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
13 2026-08-01T21:37:24Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
13 2026-08-01T21:37:26Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
13 2026-08-01T21:37:28Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
13 2026-08-01T21:37:30Z I11 PASS leaked=[]
14 2026-08-01T21:38:13Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.451371951s", "last_run": "2026-08-01T21:38:02.254099718Z", "success": true}, "enabled": true, "running": false}
14 2026-08-01T21:40:29Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
14 2026-08-01T21:40:32Z I3 PASS mentes bound_under_parent=False while absent
14 2026-08-01T21:42:45Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
14 2026-08-01T21:42:48Z I4 PASS guest init boot before=1785619357 after=1785619357 (must be equal)
14 2026-08-01T21:45:09Z I1-under-load PASS target pulled DURING a backup -> got=backup_target_absent (want backup_target_absent)
14 2026-08-01T21:47:27Z I1-under-load-recover PASS after return degraded=False target=felhom-backup
14 2026-08-01T21:49:43Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
14 2026-08-01T21:49:46Z I3 PASS adatok bound_under_parent=False while absent
14 2026-08-01T21:51:59Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
14 2026-08-01T21:52:02Z I4 PASS guest init boot before=1785619357 after=1785619357 (must be equal)
14 2026-08-01T21:52:52Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.810753555s", "last_run": "2026-08-01T21:52:51.907657536Z", "success": true}, "enabled": true, "running": false}
14 2026-08-01T21:53:51Z I7 PASS app=rallly want=C10-C014-A-235204 got=C10-C014-A-235204 snap=helyi restore_ok=True rto=29.1s
14 2026-08-01T21:54:40Z KILL-CTRL PASS controller recovered after kill
14 2026-08-01T21:55:25Z REDEPLOY PASS homebox redeployed
14 2026-08-01T21:55:28Z I5 PASS degraded=False absent_events=0 target=felhom-backup
14 2026-08-01T21:55:28Z I6 PASS healthy state must carry no degraded message; got=None
14 2026-08-01T21:55:29Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
14 2026-08-01T21:55:29Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
14 2026-08-01T21:55:31Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
14 2026-08-01T21:55:33Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
14 2026-08-01T21:55:35Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
14 2026-08-01T21:55:37Z I11 PASS leaked=[]
15 2026-08-01T21:57:55Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
15 2026-08-01T21:57:58Z I3 PASS mentes bound_under_parent=False while absent
15 2026-08-01T22:00:11Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
15 2026-08-01T22:00:13Z I4 PASS guest init boot before=1785619357 after=1785619357 (must be equal)
15 2026-08-01T22:00:43Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.939288228s", "last_run": "2026-08-01T22:00:42.599329909Z", "success": true}, "enabled": true, "running": false}
15 2026-08-01T22:02:58Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
15 2026-08-01T22:03:01Z I3 PASS adatok bound_under_parent=False while absent
15 2026-08-01T22:05:14Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
15 2026-08-01T22:05:16Z I4 PASS guest init boot before=1785619357 after=1785619357 (must be equal)
15 2026-08-01T22:06:06Z KILL-CTRL PASS controller recovered after kill
15 2026-08-01T22:06:53Z FILL-DRIVE PASS near-full (G 300M 100% /mnt/mentes) backup outcome recorded: {"db_dump": {"count": 1, "duration": "27.279924334s", "last_run": "2026-08-01T22:06:40.282160858Z", "success": true}, "enabled": true, "running": false}
15 2026-08-01T22:07:38Z REDEPLOY PASS grafana redeployed
15 2026-08-01T22:08:39Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.990582673s", "last_run": "2026-08-01T22:08:28.011005749Z", "success": true}, "enabled": true, "running": false}
15 2026-08-01T22:09:50Z I7 PASS app=rallly want=C10-C015-A-000739 got=C10-C015-A-000739 snap=helyi restore_ok=True rto=41.6s
15 2026-08-01T22:09:53Z I5 PASS degraded=False absent_events=0 target=felhom-backup
15 2026-08-01T22:09:53Z I6 PASS healthy state must carry no degraded message; got=None
15 2026-08-01T22:09:55Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
15 2026-08-01T22:09:55Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
15 2026-08-01T22:09:57Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
15 2026-08-01T22:09:59Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
15 2026-08-01T22:10:01Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
15 2026-08-01T22:10:03Z I11 PASS leaked=[]
16 2026-08-01T22:12:21Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
16 2026-08-01T22:12:24Z I3 PASS mentes bound_under_parent=False while absent
16 2026-08-01T22:14:37Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
16 2026-08-01T22:14:39Z I4 PASS guest init boot before=1785619357 after=1785619357 (must be equal)
16 2026-08-01T22:16:55Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
16 2026-08-01T22:16:58Z I3 PASS adatok bound_under_parent=False while absent
16 2026-08-01T22:19:10Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
16 2026-08-01T22:19:12Z I4 PASS guest init boot before=1785619357 after=1785619357 (must be equal)
16 2026-08-01T22:21:48Z I4-abort PASS aborted-in-place: bound=False (want False), device still present=True, opts= rw,relatime,abort,emergency_ro 0 0
16 2026-08-01T22:21:52Z I3-abort PASS calibre-web must be stopped while its namespace is aborted; running=False
16 2026-08-01T22:24:43Z I4-abort-recover PASS after re-seat: bound=True app_up=True
16 2026-08-01T22:25:32Z KILL-CTRL PASS controller recovered after kill
16 2026-08-01T22:26:14Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.998914345s", "last_run": "2026-08-01T22:26:01.683065926Z", "success": true}, "enabled": true, "running": false}
16 2026-08-01T22:26:57Z REDEPLOY PASS grafana redeployed
16 2026-08-01T22:27:46Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.784661086s", "last_run": "2026-08-01T22:27:46.541809635Z", "success": true}, "enabled": true, "running": false}
16 2026-08-01T22:28:56Z I7 PASS app=rallly want=C10-C016-A-002659 got=C10-C016-A-002659 snap=helyi restore_ok=True rto=40.9s
16 2026-08-01T22:28:59Z I5 PASS degraded=False absent_events=0 target=felhom-backup
16 2026-08-01T22:28:59Z I6 PASS healthy state must carry no degraded message; got=None
16 2026-08-01T22:29:00Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
16 2026-08-01T22:29:00Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
16 2026-08-01T22:29:02Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
16 2026-08-01T22:29:04Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
16 2026-08-01T22:29:06Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
16 2026-08-01T22:29:08Z I11 PASS leaked=[]
17 2026-08-01T22:30:45Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.559558739s", "last_run": "2026-08-01T22:30:33.408540844Z", "success": true}, "enabled": true, "running": false}
17 2026-08-01T22:30:45Z KILL-AGENT PASS agent active after kill=True; subsequent backup ok=True
17 2026-08-01T22:31:26Z BACKUP PASS {"db_dump": {"count": 1, "duration": "33.260905134s", "last_run": "2026-08-01T22:31:20.258426225Z", "success": true}, "enabled": true, "running": false}
17 2026-08-01T22:32:29Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.659629758s", "last_run": "2026-08-01T22:32:28.207890599Z", "success": true}, "enabled": true, "running": false}
17 2026-08-01T22:33:38Z I7 PASS app=rallly want=C10-C017-A-003140 got=C10-C017-A-003140 snap=helyi restore_ok=True rto=41.0s
17 2026-08-01T22:35:53Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
17 2026-08-01T22:35:56Z I3 PASS adatok bound_under_parent=False while absent
17 2026-08-01T22:38:09Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
17 2026-08-01T22:38:11Z I4 PASS guest init boot before=1785619357 after=1785619357 (must be equal)
17 2026-08-01T22:39:01Z KILL-CTRL PASS controller recovered after kill
17 2026-08-01T22:39:45Z REDEPLOY PASS grafana redeployed
17 2026-08-01T22:42:00Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
17 2026-08-01T22:42:03Z I3 PASS mentes bound_under_parent=False while absent
17 2026-08-01T22:44:16Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
17 2026-08-01T22:44:19Z I4 PASS guest init boot before=1785619357 after=1785619357 (must be equal)
17 2026-08-01T22:44:22Z I5 PASS degraded=False absent_events=0 target=felhom-backup
17 2026-08-01T22:44:22Z I6 PASS healthy state must carry no degraded message; got=None
17 2026-08-01T22:44:23Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
17 2026-08-01T22:44:23Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
17 2026-08-01T22:44:25Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
17 2026-08-01T22:44:27Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
17 2026-08-01T22:44:29Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
17 2026-08-01T22:44:32Z I11 PASS leaked=[]
18 2026-08-01T22:45:15Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.20431847s", "last_run": "2026-08-01T22:45:03.22588683Z", "success": true}, "enabled": true, "running": false}
18 2026-08-01T22:47:31Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
18 2026-08-01T22:47:33Z I3 PASS mentes bound_under_parent=False while absent
18 2026-08-01T22:49:46Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
18 2026-08-01T22:49:48Z I4 PASS guest init boot before=1785619357 after=1785619357 (must be equal)
18 2026-08-01T22:52:04Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
18 2026-08-01T22:52:07Z I3 PASS adatok bound_under_parent=False while absent
18 2026-08-01T22:54:20Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
18 2026-08-01T22:54:23Z I4 PASS guest init boot before=1785619357 after=1785619357 (must be equal)
18 2026-08-01T22:55:06Z REDEPLOY PASS grafana redeployed
18 2026-08-01T22:56:08Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.889374437s", "last_run": "2026-08-01T22:55:55.734169342Z", "success": true}, "enabled": true, "running": false}
18 2026-08-01T22:57:17Z I7 PASS app=rallly want=C10-C018-A-005508 got=C10-C018-A-005508 snap=helyi restore_ok=True rto=41.4s
18 2026-08-01T22:58:07Z KILL-CTRL PASS controller recovered after kill
18 2026-08-01T23:02:05Z HARD-RESET PASS VM returned=True apps_ready=True canaries_intact=True (a hard reset mid-write must not corrupt them)
18 2026-08-01T23:02:08Z I5 PASS degraded=False absent_events=0 target=felhom-backup
18 2026-08-01T23:02:08Z I6 PASS healthy state must carry no degraded message; got=None
18 2026-08-01T23:02:10Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
18 2026-08-01T23:02:10Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
18 2026-08-01T23:02:12Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
18 2026-08-01T23:02:14Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
18 2026-08-01T23:02:16Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
18 2026-08-01T23:02:18Z I11 PASS leaked=[]
19 2026-08-01T23:03:01Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.71036669s", "last_run": "2026-08-01T23:02:51.177403256Z", "success": true}, "enabled": true, "running": false}
19 2026-08-01T23:03:51Z KILL-CTRL PASS controller recovered after kill
19 2026-08-01T23:06:06Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
19 2026-08-01T23:06:09Z I3 PASS adatok bound_under_parent=False while absent
19 2026-08-01T23:08:22Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
19 2026-08-01T23:08:25Z I4 PASS guest init boot before=1785625268 after=1785625268 (must be equal)
19 2026-08-01T23:09:40Z REBOOT-VM PASS controller back after full VM reboot
19 2026-08-01T23:11:56Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
19 2026-08-01T23:11:59Z I3 PASS mentes bound_under_parent=False while absent
19 2026-08-01T23:14:11Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
19 2026-08-01T23:14:14Z I4 PASS guest init boot before=1785625770 after=1785625770 (must be equal)
19 2026-08-01T23:14:58Z REDEPLOY PASS grafana redeployed
19 2026-08-01T23:15:59Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.747775701s", "last_run": "2026-08-01T23:15:49.342691122Z", "success": true}, "enabled": true, "running": false}
19 2026-08-01T23:17:09Z I7 PASS app=rallly want=C10-C019-A-011500 got=C10-C019-A-011500 snap=helyi restore_ok=True rto=41.6s
19 2026-08-01T23:17:11Z I5 PASS degraded=False absent_events=0 target=felhom-backup
19 2026-08-01T23:17:11Z I6 PASS healthy state must carry no degraded message; got=None
19 2026-08-01T23:17:13Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
19 2026-08-01T23:17:13Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
19 2026-08-01T23:17:15Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
19 2026-08-01T23:17:17Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
19 2026-08-01T23:17:19Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
19 2026-08-01T23:17:21Z I11 PASS leaked=[]
20 2026-08-01T23:18:02Z REBOOT PASS controller back after guest reboot
20 2026-08-01T23:19:16Z BACKUP PASS {"db_dump": {"count": 1, "duration": "33.057982871s", "last_run": "2026-08-01T23:19:10.391093507Z", "success": true}, "enabled": true, "running": false}
20 2026-08-01T23:20:26Z I7 PASS app=rallly want=C10-C020-A-011816 got=C10-C020-A-011816 snap=helyi restore_ok=True rto=41.0s
20 2026-08-01T23:21:10Z REDEPLOY PASS grafana redeployed
20 2026-08-01T23:21:51Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.31938601s", "last_run": "2026-08-01T23:21:39.458766505Z", "success": true}, "enabled": true, "running": false}
20 2026-08-01T23:24:07Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
20 2026-08-01T23:24:10Z I3 PASS adatok bound_under_parent=False while absent
20 2026-08-01T23:26:23Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
20 2026-08-01T23:26:25Z I4 PASS guest init boot before=1785626262 after=1785626262 (must be equal)
20 2026-08-01T23:27:19Z KILL-CTRL PASS controller recovered after kill
20 2026-08-01T23:29:38Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
20 2026-08-01T23:29:40Z I3 PASS mentes bound_under_parent=False while absent
20 2026-08-01T23:31:53Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
20 2026-08-01T23:31:56Z I4 PASS guest init boot before=1785626262 after=1785626262 (must be equal)
20 2026-08-01T23:31:58Z I5 PASS degraded=False absent_events=0 target=felhom-backup
20 2026-08-01T23:31:58Z I6 PASS healthy state must carry no degraded message; got=None
20 2026-08-01T23:32:00Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
20 2026-08-01T23:32:00Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
20 2026-08-01T23:32:02Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
20 2026-08-01T23:32:04Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
20 2026-08-01T23:32:06Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
20 2026-08-01T23:32:08Z I11 PASS leaked=[]
21 2026-08-01T23:32:51Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.232854831s", "last_run": "2026-08-01T23:32:39.560692129Z", "success": true}, "enabled": true, "running": false}
21 2026-08-01T23:33:35Z REDEPLOY PASS homebox redeployed
21 2026-08-01T23:34:24Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.698974729s", "last_run": "2026-08-01T23:34:24.421847049Z", "success": true}, "enabled": true, "running": false}
21 2026-08-01T23:35:35Z I7 PASS app=rallly want=C10-C021-A-013337 got=C10-C021-A-013337 snap=helyi restore_ok=True rto=41.3s
21 2026-08-01T23:37:50Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
21 2026-08-01T23:37:53Z I3 PASS mentes bound_under_parent=False while absent
21 2026-08-01T23:40:06Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
21 2026-08-01T23:40:08Z I4 PASS guest init boot before=1785626262 after=1785626262 (must be equal)
21 2026-08-01T23:40:47Z CONCURRENCY PASS backup+restore raced; restore-status op=restore running=False (recorded, not asserted)
21 2026-08-01T23:41:36Z KILL-CTRL PASS controller recovered after kill
21 2026-08-01T23:43:52Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
21 2026-08-01T23:43:55Z I3 PASS adatok bound_under_parent=False while absent
21 2026-08-01T23:46:08Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
21 2026-08-01T23:46:10Z I4 PASS guest init boot before=1785626262 after=1785626262 (must be equal)
21 2026-08-01T23:46:12Z I5 PASS degraded=False absent_events=0 target=felhom-backup
21 2026-08-01T23:46:12Z I6 PASS healthy state must carry no degraded message; got=None
21 2026-08-01T23:46:14Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
21 2026-08-01T23:46:14Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
21 2026-08-01T23:46:16Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
21 2026-08-01T23:46:18Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
21 2026-08-01T23:46:20Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
21 2026-08-01T23:46:22Z I11 PASS leaked=[]
22 2026-08-01T23:47:08Z REDEPLOY PASS grafana redeployed
22 2026-08-01T23:49:24Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
22 2026-08-01T23:49:27Z I3 PASS mentes bound_under_parent=False while absent
22 2026-08-01T23:51:39Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
22 2026-08-01T23:51:41Z I4 PASS guest init boot before=1785626262 after=1785626262 (must be equal)
22 2026-08-01T23:54:03Z I1-under-load PASS target pulled DURING a backup -> got=backup_target_absent (want backup_target_absent)
22 2026-08-01T23:56:20Z I1-under-load-recover PASS after return degraded=False target=felhom-backup
22 2026-08-01T23:56:50Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.043140969s", "last_run": "2026-08-01T23:56:49.694291294Z", "success": true}, "enabled": true, "running": false}
22 2026-08-01T23:57:52Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.981533847s", "last_run": "2026-08-01T23:57:51.756735147Z", "success": true}, "enabled": true, "running": false}
22 2026-08-01T23:59:01Z I7 PASS app=rallly want=C10-C022-A-015704 got=C10-C022-A-015704 snap=helyi restore_ok=True rto=40.8s
22 2026-08-01T23:59:50Z KILL-CTRL PASS controller recovered after kill
22 2026-08-02T00:02:06Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
22 2026-08-02T00:02:09Z I3 PASS adatok bound_under_parent=False while absent
22 2026-08-02T00:04:22Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
22 2026-08-02T00:04:24Z I4 PASS guest init boot before=1785626262 after=1785626262 (must be equal)
22 2026-08-02T00:04:27Z I5 PASS degraded=False absent_events=0 target=felhom-backup
22 2026-08-02T00:04:27Z I6 PASS healthy state must carry no degraded message; got=None
22 2026-08-02T00:04:28Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
22 2026-08-02T00:04:28Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
22 2026-08-02T00:04:30Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
22 2026-08-02T00:04:32Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
22 2026-08-02T00:04:34Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
22 2026-08-02T00:04:36Z I11 PASS leaked=[]
23 2026-08-02T00:05:25Z FILL-DRIVE PASS near-full (G 300M 100% /mnt/mentes) backup outcome recorded: {"db_dump": {"count": 1, "duration": "27.115631074s", "last_run": "2026-08-02T00:05:12.956683735Z", "success": true}, "enabled": true, "running": false}
23 2026-08-02T00:06:15Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.744149986s", "last_run": "2026-08-02T00:06:15.151474451Z", "success": true}, "enabled": true, "running": false}
23 2026-08-02T00:07:31Z I7 PASS app=rallly want=C10-C023-A-020527 got=C10-C023-A-020527 snap=helyi restore_ok=True rto=42.8s
23 2026-08-02T00:09:48Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
23 2026-08-02T00:09:50Z I3 PASS adatok bound_under_parent=False while absent
23 2026-08-02T00:12:03Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
23 2026-08-02T00:12:07Z I4 PASS guest init boot before=1785626262 after=1785626262 (must be equal)
23 2026-08-02T00:12:56Z KILL-CTRL PASS controller recovered after kill
23 2026-08-02T00:13:41Z REDEPLOY PASS homebox redeployed
23 2026-08-02T00:15:58Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
23 2026-08-02T00:16:01Z I3 PASS mentes bound_under_parent=False while absent
23 2026-08-02T00:18:15Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
23 2026-08-02T00:18:18Z I4 PASS guest init boot before=1785626262 after=1785626262 (must be equal)
23 2026-08-02T00:18:49Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.195142894s", "last_run": "2026-08-02T00:18:47.465712394Z", "success": true}, "enabled": true, "running": false}
23 2026-08-02T00:18:52Z I5 PASS degraded=False absent_events=0 target=felhom-backup
23 2026-08-02T00:18:52Z I6 PASS healthy state must carry no degraded message; got=None
23 2026-08-02T00:18:54Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
23 2026-08-02T00:18:54Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
23 2026-08-02T00:18:56Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
23 2026-08-02T00:18:59Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
23 2026-08-02T00:19:01Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
23 2026-08-02T00:19:04Z I11 PASS leaked=[]
24 2026-08-02T00:19:49Z BACKUP PASS {"db_dump": {"count": 1, "duration": "30.825217055s", "last_run": "2026-08-02T00:19:40.099355Z", "success": true}, "enabled": true, "running": false}
24 2026-08-02T00:20:44Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.981639548s", "last_run": "2026-08-02T00:20:42.308590844Z", "success": true}, "enabled": true, "running": false}
24 2026-08-02T00:21:50Z I7 PASS app=rallly want=C10-C024-A-021951 got=C10-C024-A-021951 snap=helyi restore_ok=True rto=30.9s
24 2026-08-02T00:24:06Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
24 2026-08-02T00:24:09Z I3 PASS mentes bound_under_parent=False while absent
24 2026-08-02T00:26:22Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
24 2026-08-02T00:26:25Z I4 PASS guest init boot before=1785626262 after=1785626262 (must be equal)
24 2026-08-02T00:27:15Z KILL-CTRL PASS controller recovered after kill
24 2026-08-02T00:29:31Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
24 2026-08-02T00:29:34Z I3 PASS adatok bound_under_parent=False while absent
24 2026-08-02T00:31:47Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
24 2026-08-02T00:31:50Z I4 PASS guest init boot before=1785626262 after=1785626262 (must be equal)
24 2026-08-02T00:32:33Z REDEPLOY PASS homebox redeployed
24 2026-08-02T00:35:09Z I4-abort PASS aborted-in-place: bound=False (want False), device still present=True, opts= rw,relatime,abort,emergency_ro 0 0
24 2026-08-02T00:35:13Z I3-abort PASS calibre-web must be stopped while its namespace is aborted; running=False
24 2026-08-02T00:38:03Z I4-abort-recover PASS after re-seat: bound=True app_up=True
24 2026-08-02T00:38:06Z I5 PASS degraded=False absent_events=0 target=felhom-backup
24 2026-08-02T00:38:06Z I6 PASS healthy state must carry no degraded message; got=None
24 2026-08-02T00:38:08Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
24 2026-08-02T00:38:08Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
24 2026-08-02T00:38:10Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
24 2026-08-02T00:38:12Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
24 2026-08-02T00:38:13Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
24 2026-08-02T00:38:15Z I11 PASS leaked=[]
25 2026-08-02T00:40:33Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
25 2026-08-02T00:40:36Z I3 PASS mentes bound_under_parent=False while absent
25 2026-08-02T00:42:49Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
25 2026-08-02T00:42:52Z I4 PASS guest init boot before=1785626262 after=1785626262 (must be equal)
25 2026-08-02T00:45:08Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
25 2026-08-02T00:45:10Z I3 PASS adatok bound_under_parent=False while absent
25 2026-08-02T00:47:23Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
25 2026-08-02T00:47:26Z I4 PASS guest init boot before=1785626262 after=1785626262 (must be equal)
25 2026-08-02T00:47:55Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.646276804s", "last_run": "2026-08-02T00:47:55.027800514Z", "success": true}, "enabled": true, "running": false}
25 2026-08-02T00:48:44Z REDEPLOY PASS grafana redeployed
25 2026-08-02T00:49:53Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.717186099s", "last_run": "2026-08-02T00:49:50.193862761Z", "success": true}, "enabled": true, "running": false}
25 2026-08-02T00:51:12Z I7 PASS app=rallly want=C10-C025-A-024848 got=C10-C025-A-024848 snap=helyi restore_ok=True rto=34.4s
25 2026-08-02T00:52:41Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.051413977s", "last_run": "2026-08-02T00:52:40.717760511Z", "success": true}, "enabled": true, "running": false}
25 2026-08-02T00:52:41Z KILL-AGENT PASS agent active after kill=True; subsequent backup ok=True
25 2026-08-02T00:53:31Z KILL-CTRL PASS controller recovered after kill
25 2026-08-02T00:53:34Z I5 PASS degraded=False absent_events=0 target=felhom-backup
25 2026-08-02T00:53:34Z I6 PASS healthy state must carry no degraded message; got=None
25 2026-08-02T00:53:36Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
25 2026-08-02T00:53:36Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
25 2026-08-02T00:53:37Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
25 2026-08-02T00:53:39Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
25 2026-08-02T00:53:41Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
25 2026-08-02T00:53:44Z I11 PASS leaked=[]
26 2026-08-02T00:54:27Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.067773015s", "last_run": "2026-08-02T00:54:15.396183229Z", "success": true}, "enabled": true, "running": false}
26 2026-08-02T00:55:16Z KILL-CTRL PASS controller recovered after kill
26 2026-08-02T00:57:32Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
26 2026-08-02T00:57:35Z I3 PASS mentes bound_under_parent=False while absent
26 2026-08-02T00:59:48Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
26 2026-08-02T00:59:50Z I4 PASS guest init boot before=1785626262 after=1785626262 (must be equal)
26 2026-08-02T01:00:51Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.960624703s", "last_run": "2026-08-02T01:00:39.950275Z", "success": true}, "enabled": true, "running": false}
26 2026-08-02T01:02:01Z I7 PASS app=rallly want=C10-C026-A-025952 got=C10-C026-A-025952 snap=helyi restore_ok=True rto=41.4s
26 2026-08-02T01:15:31Z HARD-RESET VIOLATION VM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
26 2026-08-02T01:17:47Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
26 2026-08-02T01:17:51Z I3 PASS adatok bound_under_parent=False while absent
26 2026-08-02T01:20:04Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
26 2026-08-02T01:20:06Z I4 PASS guest init boot before=1785632702 after=1785632702 (must be equal)
26 2026-08-02T01:20:50Z REDEPLOY PASS grafana redeployed
26 2026-08-02T01:20:53Z I5 PASS degraded=False absent_events=0 target=felhom-backup
26 2026-08-02T01:20:53Z I6 PASS healthy state must carry no degraded message; got=None
26 2026-08-02T01:20:55Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
26 2026-08-02T01:20:55Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
26 2026-08-02T01:20:57Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
26 2026-08-02T01:20:58Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
26 2026-08-02T01:21:00Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
26 2026-08-02T01:21:02Z I11 PASS leaked=[]
27 2026-08-02T01:21:33Z BACKUP PASS {"db_dump": {"count": 1, "duration": "17.871195759s", "last_run": "2026-08-02T01:21:24.676991231Z", "success": true}, "enabled": true, "running": false}
27 2026-08-02T01:22:48Z REBOOT-VM PASS controller back after full VM reboot
27 2026-08-02T01:25:04Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
27 2026-08-02T01:25:07Z I3 PASS adatok bound_under_parent=False while absent
27 2026-08-02T01:27:20Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
27 2026-08-02T01:27:22Z I4 PASS guest init boot before=1785633757 after=1785633757 (must be equal)
27 2026-08-02T01:28:06Z REDEPLOY PASS homebox redeployed
27 2026-08-02T01:30:21Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
27 2026-08-02T01:30:23Z I3 PASS mentes bound_under_parent=False while absent
27 2026-08-02T01:32:36Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
27 2026-08-02T01:32:39Z I4 PASS guest init boot before=1785633757 after=1785633757 (must be equal)
27 2026-08-02T01:33:28Z KILL-CTRL PASS controller recovered after kill
27 2026-08-02T01:34:18Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.733119419s", "last_run": "2026-08-02T01:34:18.829727455Z", "success": true}, "enabled": true, "running": false}
27 2026-08-02T01:35:29Z I7 PASS app=rallly want=C10-C027-A-033330 got=C10-C027-A-033330 snap=helyi restore_ok=True rto=41.0s
27 2026-08-02T01:35:31Z I5 PASS degraded=False absent_events=0 target=felhom-backup
27 2026-08-02T01:35:31Z I6 PASS healthy state must carry no degraded message; got=None
27 2026-08-02T01:35:33Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
27 2026-08-02T01:35:33Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
27 2026-08-02T01:35:35Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
27 2026-08-02T01:35:37Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
27 2026-08-02T01:35:39Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
27 2026-08-02T01:35:41Z I11 PASS leaked=[]
28 2026-08-02T01:37:59Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
28 2026-08-02T01:38:02Z I3 PASS adatok bound_under_parent=False while absent
28 2026-08-02T01:40:15Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
28 2026-08-02T01:40:17Z I4 PASS guest init boot before=1785633757 after=1785633757 (must be equal)
28 2026-08-02T01:40:47Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.038226291s", "last_run": "2026-08-02T01:40:46.835262516Z", "success": true}, "enabled": true, "running": false}
28 2026-08-02T01:41:30Z REDEPLOY PASS homebox redeployed
28 2026-08-02T01:43:46Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
28 2026-08-02T01:43:49Z I3 PASS mentes bound_under_parent=False while absent
28 2026-08-02T01:46:02Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
28 2026-08-02T01:46:04Z I4 PASS guest init boot before=1785633757 after=1785633757 (must be equal)
28 2026-08-02T01:47:06Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.137352809s", "last_run": "2026-08-02T01:46:53.969942392Z", "success": true}, "enabled": true, "running": false}
28 2026-08-02T01:48:17Z I7 PASS app=rallly want=C10-C028-A-034606 got=C10-C028-A-034606 snap=helyi restore_ok=True rto=41.8s
28 2026-08-02T01:49:06Z KILL-CTRL PASS controller recovered after kill
28 2026-08-02T01:49:45Z REBOOT PASS controller back after guest reboot
28 2026-08-02T01:49:48Z I5 PASS degraded=False absent_events=0 target=felhom-backup
28 2026-08-02T01:49:48Z I6 PASS healthy state must carry no degraded message; got=None
28 2026-08-02T01:49:50Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
28 2026-08-02T01:49:50Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
28 2026-08-02T01:49:52Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
28 2026-08-02T01:49:54Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
28 2026-08-02T01:49:56Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
28 2026-08-02T01:49:58Z I11 PASS leaked=[]
29 2026-08-02T01:50:50Z CONCURRENCY PASS backup+restore raced; restore-status op=None running=False (recorded, not asserted)
29 2026-08-02T01:51:39Z KILL-CTRL PASS controller recovered after kill
29 2026-08-02T01:52:21Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.616661678s", "last_run": "2026-08-02T01:52:09.401481344Z", "success": true}, "enabled": true, "running": false}
29 2026-08-02T01:54:37Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
29 2026-08-02T01:54:40Z I3 PASS adatok bound_under_parent=False while absent
29 2026-08-02T01:56:53Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
29 2026-08-02T01:56:55Z I4 PASS guest init boot before=1785635365 after=1785635365 (must be equal)
29 2026-08-02T01:57:44Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.049492517s", "last_run": "2026-08-02T01:57:44.096826434Z", "success": true}, "enabled": true, "running": false}
29 2026-08-02T01:58:53Z I7 PASS app=rallly want=C10-C029-A-035657 got=C10-C029-A-035657 snap=helyi restore_ok=True rto=41.2s
29 2026-08-02T01:59:37Z REDEPLOY PASS grafana redeployed
29 2026-08-02T02:01:53Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
29 2026-08-02T02:01:55Z I3 PASS mentes bound_under_parent=False while absent
29 2026-08-02T02:04:08Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
29 2026-08-02T02:04:10Z I4 PASS guest init boot before=1785635365 after=1785635365 (must be equal)
29 2026-08-02T02:04:13Z I5 PASS degraded=False absent_events=0 target=felhom-backup
29 2026-08-02T02:04:13Z I6 PASS healthy state must carry no degraded message; got=None
29 2026-08-02T02:04:15Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
29 2026-08-02T02:04:15Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
29 2026-08-02T02:04:16Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
29 2026-08-02T02:04:18Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
29 2026-08-02T02:04:20Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
29 2026-08-02T02:04:22Z I11 PASS leaked=[]
30 2026-08-02T02:05:08Z REDEPLOY PASS grafana redeployed
30 2026-08-02T02:07:28Z I1-under-load PASS target pulled DURING a backup -> got=backup_target_absent (want backup_target_absent)
30 2026-08-02T02:09:46Z I1-under-load-recover PASS after return degraded=False target=felhom-backup
30 2026-08-02T02:10:35Z KILL-CTRL PASS controller recovered after kill
30 2026-08-02T02:12:51Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
30 2026-08-02T02:12:53Z I3 PASS mentes bound_under_parent=False while absent
30 2026-08-02T02:15:06Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
30 2026-08-02T02:15:09Z I4 PASS guest init boot before=1785635365 after=1785635365 (must be equal)
30 2026-08-02T02:15:38Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.497470443s", "last_run": "2026-08-02T02:15:38.593694037Z", "success": true}, "enabled": true, "running": false}
30 2026-08-02T02:17:54Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
30 2026-08-02T02:17:56Z I3 PASS adatok bound_under_parent=False while absent
30 2026-08-02T02:20:09Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
30 2026-08-02T02:20:12Z I4 PASS guest init boot before=1785635365 after=1785635365 (must be equal)
30 2026-08-02T02:21:13Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.992215956s", "last_run": "2026-08-02T02:21:01.139466456Z", "success": true}, "enabled": true, "running": false}
30 2026-08-02T02:22:22Z I7 PASS app=rallly want=C10-C030-A-042014 got=C10-C030-A-042014 snap=helyi restore_ok=True rto=41.2s
30 2026-08-02T02:22:25Z I5 PASS degraded=False absent_events=0 target=felhom-backup
30 2026-08-02T02:22:25Z I6 PASS healthy state must carry no degraded message; got=None
30 2026-08-02T02:22:27Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
30 2026-08-02T02:22:27Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
30 2026-08-02T02:22:29Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
30 2026-08-02T02:22:31Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
30 2026-08-02T02:22:33Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
30 2026-08-02T02:22:35Z I11 PASS leaked=[]
31 2026-08-02T02:23:18Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.13226777s", "last_run": "2026-08-02T02:23:06.357622568Z", "success": true}, "enabled": true, "running": false}
31 2026-08-02T02:24:20Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.027054408s", "last_run": "2026-08-02T02:24:07.807209046Z", "success": true}, "enabled": true, "running": false}
31 2026-08-02T02:25:29Z I7 PASS app=rallly want=C10-C031-A-042320 got=C10-C031-A-042320 snap=helyi restore_ok=True rto=41.4s
31 2026-08-02T02:26:18Z KILL-CTRL PASS controller recovered after kill
31 2026-08-02T02:27:05Z FILL-DRIVE PASS near-full (G 300M 100% /mnt/mentes) backup outcome recorded: {"db_dump": {"count": 1, "duration": "27.759263074s", "last_run": "2026-08-02T02:26:53.327742958Z", "success": true}, "enabled": true, "running": false}
31 2026-08-02T02:29:21Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
31 2026-08-02T02:29:24Z I3 PASS adatok bound_under_parent=False while absent
31 2026-08-02T02:31:37Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
31 2026-08-02T02:31:40Z I4 PASS guest init boot before=1785635365 after=1785635365 (must be equal)
31 2026-08-02T02:32:23Z REDEPLOY PASS grafana redeployed
31 2026-08-02T02:34:39Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
31 2026-08-02T02:34:42Z I3 PASS mentes bound_under_parent=False while absent
31 2026-08-02T02:36:55Z I1-pair VIOLATION mentes return -> got=None sev=None (want backup_target_restored/info)
31 2026-08-02T02:36:57Z I4 PASS guest init boot before=1785635365 after=1785635365 (must be equal)
31 2026-08-02T02:37:00Z I5 PASS degraded=False absent_events=0 target=felhom-backup
31 2026-08-02T02:37:00Z I6 PASS healthy state must carry no degraded message; got=None
31 2026-08-02T02:37:01Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
31 2026-08-02T02:37:01Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
31 2026-08-02T02:37:03Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
31 2026-08-02T02:37:05Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
31 2026-08-02T02:37:07Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
31 2026-08-02T02:37:09Z I11 PASS leaked=[]
32 2026-08-02T02:39:27Z I1 VIOLATION mentes absent -> got=None sev=None (want backup_target_absent/error)
32 2026-08-02T02:39:29Z I3 PASS mentes bound_under_parent=False while absent
32 2026-08-02T02:41:42Z I1-pair VIOLATION mentes return -> got=None sev=None (want backup_target_restored/info)
32 2026-08-02T02:41:45Z I4 PASS guest init boot before=1785635365 after=1785635365 (must be equal)
32 2026-08-02T02:44:00Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
32 2026-08-02T02:44:03Z I3 PASS adatok bound_under_parent=False while absent
32 2026-08-02T02:46:16Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
32 2026-08-02T02:46:18Z I4 PASS guest init boot before=1785635365 after=1785635365 (must be equal)
32 2026-08-02T02:46:59Z BACKUP PASS {"db_dump": {"count": 1, "duration": "29.460511127s", "last_run": "2026-08-02T02:46:49.295016477Z", "success": true}, "enabled": true, "running": false}
32 2026-08-02T02:47:43Z REDEPLOY PASS grafana redeployed
32 2026-08-02T02:50:23Z I4-abort PASS aborted-in-place: bound=False (want False), device still present=True, opts= rw,relatime,abort,emergency_ro 0 0
32 2026-08-02T02:50:27Z I3-abort PASS calibre-web must be stopped while its namespace is aborted; running=False
32 2026-08-02T02:53:17Z I4-abort-recover PASS after re-seat: bound=True app_up=True
32 2026-08-02T02:54:06Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.776116701s", "last_run": "2026-08-02T02:54:06.381587836Z", "success": true}, "enabled": true, "running": false}
32 2026-08-02T02:55:16Z I7 PASS app=rallly want=C10-C032-A-045319 got=C10-C032-A-045319 snap=helyi restore_ok=True rto=41.4s
32 2026-08-02T02:56:06Z KILL-CTRL PASS controller recovered after kill
32 2026-08-02T02:56:09Z I5 PASS degraded=False absent_events=0 target=felhom-backup
32 2026-08-02T02:56:09Z I6 PASS healthy state must carry no degraded message; got=None
32 2026-08-02T02:56:11Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
32 2026-08-02T02:56:11Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
32 2026-08-02T02:56:12Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
32 2026-08-02T02:56:14Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
32 2026-08-02T02:56:16Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
32 2026-08-02T02:56:18Z I11 PASS leaked=[]
33 2026-08-02T02:57:10Z KILL-CTRL PASS controller recovered after kill
33 2026-08-02T02:58:34Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.939510294s", "last_run": "2026-08-02T02:58:33.660056025Z", "success": true}, "enabled": true, "running": false}
33 2026-08-02T02:58:34Z KILL-AGENT PASS agent active after kill=True; subsequent backup ok=True
33 2026-08-02T02:59:36Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.215228172s", "last_run": "2026-08-02T02:59:36.103774656Z", "success": true}, "enabled": true, "running": false}
33 2026-08-02T03:00:45Z I7 PASS app=rallly want=C10-C033-A-045848 got=C10-C033-A-045848 snap=helyi restore_ok=True rto=40.7s
33 2026-08-02T03:01:29Z REDEPLOY VIOLATION grafana redeployed
33 2026-08-02T03:03:46Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
33 2026-08-02T03:03:48Z I3 PASS adatok bound_under_parent=False while absent
33 2026-08-02T03:06:00Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
33 2026-08-02T03:06:03Z I4 PASS guest init boot before=1785635365 after=1785635365 (must be equal)
33 2026-08-02T03:08:18Z I1 VIOLATION mentes absent -> got=None sev=None (want backup_target_absent/error)
33 2026-08-02T03:08:21Z I3 PASS mentes bound_under_parent=False while absent
33 2026-08-02T03:10:34Z I1-pair VIOLATION mentes return -> got=None sev=None (want backup_target_restored/info)
33 2026-08-02T03:10:37Z I4 PASS guest init boot before=1785635365 after=1785635365 (must be equal)
33 2026-08-02T03:11:06Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.135287499s", "last_run": "2026-08-02T03:11:06.220816081Z", "success": true}, "enabled": true, "running": false}
33 2026-08-02T03:11:09Z I5 PASS degraded=False absent_events=0 target=felhom-backup
33 2026-08-02T03:11:09Z I6 PASS healthy state must carry no degraded message; got=None
33 2026-08-02T03:11:11Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
33 2026-08-02T03:11:11Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
33 2026-08-02T03:11:12Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
33 2026-08-02T03:11:15Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
33 2026-08-02T03:11:16Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
33 2026-08-02T03:11:18Z I11 PASS leaked=[]
34 2026-08-02T03:24:56Z HARD-RESET VIOLATION VM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
34 2026-08-02T03:25:40Z REDEPLOY PASS grafana redeployed
34 2026-08-02T03:26:42Z BACKUP PASS {"db_dump": {"count": 1, "duration": "30.123463849s", "last_run": "2026-08-02T03:26:33.352940308Z", "success": true}, "enabled": true, "running": false}
34 2026-08-02T03:27:53Z I7 PASS app=rallly want=C10-C034-A-052542 got=C10-C034-A-052542 snap=helyi restore_ok=True rto=41.6s
34 2026-08-02T03:30:08Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
34 2026-08-02T03:30:11Z I3 PASS adatok bound_under_parent=False while absent
34 2026-08-02T03:32:24Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
34 2026-08-02T03:32:27Z I4 PASS guest init boot before=1785640463 after=1785640463 (must be equal)
34 2026-08-02T03:33:16Z KILL-CTRL PASS controller recovered after kill
34 2026-08-02T03:33:58Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.493828931s", "last_run": "2026-08-02T03:33:46.251686967Z", "success": true}, "enabled": true, "running": false}
34 2026-08-02T03:36:14Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
34 2026-08-02T03:36:16Z I3 PASS mentes bound_under_parent=False while absent
34 2026-08-02T03:38:28Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
34 2026-08-02T03:38:31Z I4 PASS guest init boot before=1785640463 after=1785640463 (must be equal)
34 2026-08-02T03:38:33Z I5 PASS degraded=False absent_events=0 target=felhom-backup
34 2026-08-02T03:38:33Z I6 PASS healthy state must carry no degraded message; got=None
34 2026-08-02T03:38:35Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
34 2026-08-02T03:38:35Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
34 2026-08-02T03:38:37Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
34 2026-08-02T03:38:39Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
34 2026-08-02T03:38:41Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
34 2026-08-02T03:38:43Z I11 PASS leaked=[]
35 2026-08-02T03:39:46Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.312024883s", "last_run": "2026-08-02T03:39:34.299162032Z", "success": true}, "enabled": true, "running": false}
35 2026-08-02T03:40:56Z I7 PASS app=rallly want=C10-C035-A-053847 got=C10-C035-A-053847 snap=helyi restore_ok=True rto=41.4s
35 2026-08-02T03:41:39Z REDEPLOY PASS grafana redeployed
35 2026-08-02T03:42:28Z KILL-CTRL PASS controller recovered after kill
35 2026-08-02T03:44:44Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
35 2026-08-02T03:44:47Z I3 PASS adatok bound_under_parent=False while absent
35 2026-08-02T03:47:00Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
35 2026-08-02T03:47:02Z I4 PASS guest init boot before=1785640463 after=1785640463 (must be equal)
35 2026-08-02T03:49:17Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
35 2026-08-02T03:49:20Z I3 PASS mentes bound_under_parent=False while absent
35 2026-08-02T03:51:34Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
35 2026-08-02T03:51:36Z I4 PASS guest init boot before=1785640463 after=1785640463 (must be equal)
35 2026-08-02T03:52:17Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.843850494s", "last_run": "2026-08-02T03:52:05.835797124Z", "success": true}, "enabled": true, "running": false}
35 2026-08-02T03:53:33Z REBOOT-VM PASS controller back after full VM reboot
35 2026-08-02T03:53:36Z I5 PASS degraded=False absent_events=0 target=felhom-backup
35 2026-08-02T03:53:36Z I6 PASS healthy state must carry no degraded message; got=None
35 2026-08-02T03:53:38Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
35 2026-08-02T03:53:38Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
35 2026-08-02T03:53:40Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
35 2026-08-02T03:53:42Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
35 2026-08-02T03:53:44Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
35 2026-08-02T03:53:47Z I11 PASS leaked=[]
36 2026-08-02T03:56:04Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
36 2026-08-02T03:56:07Z I3 PASS adatok bound_under_parent=False while absent
36 2026-08-02T03:58:20Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
36 2026-08-02T03:58:22Z I4 PASS guest init boot before=1785642801 after=1785642801 (must be equal)
36 2026-08-02T03:59:06Z REDEPLOY PASS grafana redeployed
36 2026-08-02T04:00:06Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.362006073s", "last_run": "2026-08-02T03:59:56.205720792Z", "success": true}, "enabled": true, "running": false}
36 2026-08-02T04:01:15Z I7 PASS app=rallly want=C10-C036-A-055908 got=C10-C036-A-055908 snap=helyi restore_ok=True rto=41.4s
36 2026-08-02T04:02:05Z KILL-CTRL PASS controller recovered after kill
36 2026-08-02T04:04:20Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
36 2026-08-02T04:04:24Z I3 PASS mentes bound_under_parent=False while absent
36 2026-08-02T04:06:36Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
36 2026-08-02T04:06:39Z I4 PASS guest init boot before=1785642801 after=1785642801 (must be equal)
36 2026-08-02T04:07:21Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.746272981s", "last_run": "2026-08-02T04:07:09.192531609Z", "success": true}, "enabled": true, "running": false}
36 2026-08-02T04:07:59Z REBOOT PASS controller back after guest reboot
36 2026-08-02T04:08:02Z I5 PASS degraded=False absent_events=0 target=felhom-backup
36 2026-08-02T04:08:02Z I6 PASS healthy state must carry no degraded message; got=None
36 2026-08-02T04:08:04Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
36 2026-08-02T04:08:04Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
36 2026-08-02T04:08:06Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
36 2026-08-02T04:08:08Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
36 2026-08-02T04:08:09Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
36 2026-08-02T04:08:12Z I11 PASS leaked=[]
37 2026-08-02T04:09:03Z KILL-CTRL PASS controller recovered after kill
37 2026-08-02T04:11:18Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
37 2026-08-02T04:11:21Z I3 PASS adatok bound_under_parent=False while absent
37 2026-08-02T04:13:34Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
37 2026-08-02T04:13:36Z I4 PASS guest init boot before=1785643659 after=1785643659 (must be equal)
37 2026-08-02T04:15:52Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
37 2026-08-02T04:15:54Z I3 PASS mentes bound_under_parent=False while absent
37 2026-08-02T04:18:07Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
37 2026-08-02T04:18:09Z I4 PASS guest init boot before=1785643659 after=1785643659 (must be equal)
37 2026-08-02T04:19:10Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.310789623s", "last_run": "2026-08-02T04:18:59.857398037Z", "success": true}, "enabled": true, "running": false}
37 2026-08-02T04:20:20Z I7 PASS app=rallly want=C10-C037-A-061811 got=C10-C037-A-061811 snap=helyi restore_ok=True rto=41.5s
37 2026-08-02T04:20:58Z CONCURRENCY PASS backup+restore raced; restore-status op=restore running=False (recorded, not asserted)
37 2026-08-02T04:21:41Z REDEPLOY PASS grafana redeployed
37 2026-08-02T04:22:22Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.537656045s", "last_run": "2026-08-02T04:22:11.330859937Z", "success": true}, "enabled": true, "running": false}
37 2026-08-02T04:22:25Z I5 PASS degraded=False absent_events=0 target=felhom-backup
37 2026-08-02T04:22:25Z I6 PASS healthy state must carry no degraded message; got=None
37 2026-08-02T04:22:26Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
37 2026-08-02T04:22:26Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
37 2026-08-02T04:22:28Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
37 2026-08-02T04:22:30Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
37 2026-08-02T04:22:32Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
37 2026-08-02T04:22:34Z I11 PASS leaked=[]
38 2026-08-02T04:23:20Z REDEPLOY PASS grafana redeployed
38 2026-08-02T04:25:36Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
38 2026-08-02T04:25:39Z I3 PASS adatok bound_under_parent=False while absent
38 2026-08-02T04:27:52Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
38 2026-08-02T04:27:54Z I4 PASS guest init boot before=1785643659 after=1785643659 (must be equal)
38 2026-08-02T04:28:56Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.521112866s", "last_run": "2026-08-02T04:28:44.25193983Z", "success": true}, "enabled": true, "running": false}
38 2026-08-02T04:30:05Z I7 PASS app=rallly want=C10-C038-A-062756 got=C10-C038-A-062756 snap=helyi restore_ok=True rto=41.6s
38 2026-08-02T04:32:21Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
38 2026-08-02T04:32:24Z I3 PASS mentes bound_under_parent=False while absent
38 2026-08-02T04:34:37Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
38 2026-08-02T04:34:39Z I4 PASS guest init boot before=1785643659 after=1785643659 (must be equal)
38 2026-08-02T04:37:00Z I1-under-load PASS target pulled DURING a backup -> got=backup_target_absent (want backup_target_absent)
38 2026-08-02T04:39:18Z I1-under-load-recover PASS after return degraded=False target=felhom-backup
38 2026-08-02T04:40:07Z KILL-CTRL PASS controller recovered after kill
38 2026-08-02T04:40:37Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.153659203s", "last_run": "2026-08-02T04:40:36.916754149Z", "success": true}, "enabled": true, "running": false}
38 2026-08-02T04:40:39Z I5 PASS degraded=False absent_events=0 target=felhom-backup
38 2026-08-02T04:40:39Z I6 PASS healthy state must carry no degraded message; got=None
38 2026-08-02T04:40:41Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
38 2026-08-02T04:40:41Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
38 2026-08-02T04:40:43Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
38 2026-08-02T04:40:45Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
38 2026-08-02T04:40:47Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
38 2026-08-02T04:40:50Z I11 PASS leaked=[]
39 2026-08-02T04:41:34Z BACKUP PASS {"db_dump": {"count": 1, "duration": "33.782651419s", "last_run": "2026-08-02T04:41:27.965253842Z", "success": true}, "enabled": true, "running": false}
39 2026-08-02T04:42:23Z KILL-CTRL PASS controller recovered after kill
39 2026-08-02T04:43:13Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.848003001s", "last_run": "2026-08-02T04:43:13.643818435Z", "success": true}, "enabled": true, "running": false}
39 2026-08-02T04:44:25Z I7 PASS app=rallly want=C10-C039-A-064225 got=C10-C039-A-064225 snap=helyi restore_ok=True rto=41.6s
39 2026-08-02T04:46:41Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
39 2026-08-02T04:46:44Z I3 PASS mentes bound_under_parent=False while absent
39 2026-08-02T04:48:57Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
39 2026-08-02T04:49:00Z I4 PASS guest init boot before=1785643659 after=1785643659 (must be equal)
39 2026-08-02T04:49:52Z FILL-DRIVE PASS near-full (G 288M 100% /mnt/mentes) backup outcome recorded: {"db_dump": {"count": 1, "duration": "31.31928094s", "last_run": "2026-08-02T04:49:38.823414524Z", "success": true}, "enabled": true, "running": false}
39 2026-08-02T04:52:12Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
39 2026-08-02T04:52:16Z I3 PASS adatok bound_under_parent=False while absent
39 2026-08-02T04:54:30Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
39 2026-08-02T04:54:32Z I4 PASS guest init boot before=1785643659 after=1785643659 (must be equal)
39 2026-08-02T04:55:16Z REDEPLOY PASS grafana redeployed
39 2026-08-02T04:55:19Z I5 PASS degraded=False absent_events=0 target=felhom-backup
39 2026-08-02T04:55:19Z I6 PASS healthy state must carry no degraded message; got=None
39 2026-08-02T04:55:21Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
39 2026-08-02T04:55:21Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
39 2026-08-02T04:55:23Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
39 2026-08-02T04:55:25Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
39 2026-08-02T04:55:27Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
39 2026-08-02T04:55:29Z I11 PASS leaked=[]
Can't render this file because it contains an unexpected character in line 6 and column 37.
documentation/tests/campaign10-evidence-2026-07-31/state/rto.tsv
+38
View File
@@ -0,0 +1,38 @@
1 2026-08-01T18:00:35Z rallly tier1 41.9 True
2 2026-08-01T18:19:37Z rallly tier1 41.6 True
3 2026-08-01T18:39:14Z rallly tier1 45.0 True
4 2026-08-01T18:48:31Z rallly tier1 42.1 True
5 2026-08-01T19:03:46Z rallly tier1 42.1 True
6 2026-08-01T19:11:26Z rallly tier1 41.6 True
7 2026-08-01T19:31:20Z rallly tier1 41.4 True
8 2026-08-01T19:45:27Z rallly tier1 41.4 True
9 2026-08-01T20:15:13Z rallly tier1 41.9 True
11 2026-08-01T21:06:54Z rallly tier1 49.6 True
12 2026-08-01T21:22:18Z rallly tier1 41.4 True
13 2026-08-01T21:32:43Z rallly tier1 29.1 True
14 2026-08-01T21:53:51Z rallly tier1 29.1 True
15 2026-08-01T22:09:50Z rallly tier1 41.6 True
16 2026-08-01T22:28:56Z rallly tier1 40.9 True
17 2026-08-01T22:33:38Z rallly tier1 41.0 True
18 2026-08-01T22:57:17Z rallly tier1 41.4 True
19 2026-08-01T23:17:09Z rallly tier1 41.6 True
20 2026-08-01T23:20:26Z rallly tier1 41.0 True
21 2026-08-01T23:35:35Z rallly tier1 41.3 True
22 2026-08-01T23:59:01Z rallly tier1 40.8 True
23 2026-08-02T00:07:31Z rallly tier1 42.8 True
24 2026-08-02T00:21:50Z rallly tier1 30.9 True
25 2026-08-02T00:51:12Z rallly tier1 34.4 True
26 2026-08-02T01:02:01Z rallly tier1 41.4 True
27 2026-08-02T01:35:29Z rallly tier1 41.0 True
28 2026-08-02T01:48:17Z rallly tier1 41.8 True
29 2026-08-02T01:58:53Z rallly tier1 41.2 True
30 2026-08-02T02:22:22Z rallly tier1 41.2 True
31 2026-08-02T02:25:29Z rallly tier1 41.4 True
32 2026-08-02T02:55:16Z rallly tier1 41.4 True
33 2026-08-02T03:00:45Z rallly tier1 40.7 True
34 2026-08-02T03:27:53Z rallly tier1 41.6 True
35 2026-08-02T03:40:56Z rallly tier1 41.4 True
36 2026-08-02T04:01:15Z rallly tier1 41.4 True
37 2026-08-02T04:20:20Z rallly tier1 41.5 True
38 2026-08-02T04:30:05Z rallly tier1 41.6 True
39 2026-08-02T04:44:25Z rallly tier1 41.6 True
1 1 2026-08-01T18:00:35Z rallly tier1 41.9 True
2 2 2026-08-01T18:19:37Z rallly tier1 41.6 True
3 3 2026-08-01T18:39:14Z rallly tier1 45.0 True
4 4 2026-08-01T18:48:31Z rallly tier1 42.1 True
5 5 2026-08-01T19:03:46Z rallly tier1 42.1 True
6 6 2026-08-01T19:11:26Z rallly tier1 41.6 True
7 7 2026-08-01T19:31:20Z rallly tier1 41.4 True
8 8 2026-08-01T19:45:27Z rallly tier1 41.4 True
9 9 2026-08-01T20:15:13Z rallly tier1 41.9 True
10 11 2026-08-01T21:06:54Z rallly tier1 49.6 True
11 12 2026-08-01T21:22:18Z rallly tier1 41.4 True
12 13 2026-08-01T21:32:43Z rallly tier1 29.1 True
13 14 2026-08-01T21:53:51Z rallly tier1 29.1 True
14 15 2026-08-01T22:09:50Z rallly tier1 41.6 True
15 16 2026-08-01T22:28:56Z rallly tier1 40.9 True
16 17 2026-08-01T22:33:38Z rallly tier1 41.0 True
17 18 2026-08-01T22:57:17Z rallly tier1 41.4 True
18 19 2026-08-01T23:17:09Z rallly tier1 41.6 True
19 20 2026-08-01T23:20:26Z rallly tier1 41.0 True
20 21 2026-08-01T23:35:35Z rallly tier1 41.3 True
21 22 2026-08-01T23:59:01Z rallly tier1 40.8 True
22 23 2026-08-02T00:07:31Z rallly tier1 42.8 True
23 24 2026-08-02T00:21:50Z rallly tier1 30.9 True
24 25 2026-08-02T00:51:12Z rallly tier1 34.4 True
25 26 2026-08-02T01:02:01Z rallly tier1 41.4 True
26 27 2026-08-02T01:35:29Z rallly tier1 41.0 True
27 28 2026-08-02T01:48:17Z rallly tier1 41.8 True
28 29 2026-08-02T01:58:53Z rallly tier1 41.2 True
29 30 2026-08-02T02:22:22Z rallly tier1 41.2 True
30 31 2026-08-02T02:25:29Z rallly tier1 41.4 True
31 32 2026-08-02T02:55:16Z rallly tier1 41.4 True
32 33 2026-08-02T03:00:45Z rallly tier1 40.7 True
33 34 2026-08-02T03:27:53Z rallly tier1 41.6 True
34 35 2026-08-02T03:40:56Z rallly tier1 41.4 True
35 36 2026-08-02T04:01:15Z rallly tier1 41.4 True
36 37 2026-08-02T04:20:20Z rallly tier1 41.5 True
37 38 2026-08-02T04:30:05Z rallly tier1 41.6 True
38 39 2026-08-02T04:44:25Z rallly tier1 41.6 True
documentation/tests/campaign10-evidence-2026-07-31/state/runner.log
+96
View File
@@ -1,2 +1,98 @@
19:51:13 === Campaign 10 Phase B start — max 45 cycles, deadline 06:51 ===
19:51:15 cycle 1: detach_nontarget -> kill_agent_mid_backup -> backup -> restore_verify -> detach_target -> redeploy_app -> kill_controller
20:06:56 cycle 1 done — violations so far: 0
20:06:58 cycle 2: redeploy_app -> detach_nontarget -> kill_controller -> backup -> hard_reset_mid_write -> restore_verify -> detach_target
20:24:24 cycle 2 done — violations so far: 0
20:24:26 cycle 3: backup -> redeploy_app -> detach_target -> reboot_vm -> detach_nontarget -> kill_controller -> restore_verify
20:39:28 cycle 3 done — violations so far: 0
20:39:30 cycle 4: detach_nontarget -> redeploy_app -> backup -> reboot_guest -> restore_verify -> detach_target -> kill_controller
20:54:07 cycle 4 done — violations so far: 0
20:54:09 cycle 5: detach_nontarget -> backup -> concurrent_backup_restore -> redeploy_app -> kill_controller -> restore_verify -> detach_target
21:08:49 cycle 5 done — violations so far: 0
21:08:59 cycle 6: restore_verify -> detach_target -> backup -> concurrent_backup_detach -> kill_controller -> redeploy_app -> detach_nontarget
21:27:40 cycle 6 done — violations so far: 0
21:27:42 cycle 7: redeploy_app -> kill_controller -> restore_verify -> fill_drive -> detach_target -> detach_nontarget -> backup
21:41:48 cycle 7 done — violations so far: 0
21:41:50 cycle 8: redeploy_app -> kill_controller -> restore_verify -> backup -> detach_target -> abort_fs_inplace -> detach_nontarget
22:00:49 cycle 8 done — violations so far: 0
22:00:51 cycle 9: redeploy_app -> kill_controller -> kill_agent_mid_backup -> detach_nontarget -> detach_target -> restore_verify -> backup
22:15:56 cycle 9 done — violations so far: 0
22:15:58 cycle 10: detach_target -> detach_nontarget -> hard_reset_mid_write -> redeploy_app -> restore_verify -> backup -> kill_controller
22:38:40 !! HARD-RESET VIOLATION: VM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
22:51:05 cycle 10 done — violations so far: 1
22:51:07 cycle 11: kill_controller -> reboot_vm -> backup -> detach_target -> detach_nontarget -> restore_verify -> redeploy_app
23:08:30 cycle 11 done — violations so far: 1
23:08:34 cycle 12: backup -> redeploy_app -> detach_nontarget -> kill_controller -> detach_target -> restore_verify -> reboot_guest
23:23:11 cycle 12 done — violations so far: 1
23:23:13 cycle 13: redeploy_app -> backup -> kill_controller -> detach_target -> concurrent_backup_restore -> restore_verify -> detach_nontarget
23:37:30 cycle 13 done — violations so far: 1
23:37:32 cycle 14: backup -> detach_target -> concurrent_backup_detach -> detach_nontarget -> restore_verify -> kill_controller -> redeploy_app
23:55:37 cycle 14 done — violations so far: 1
23:55:39 cycle 15: detach_target -> backup -> detach_nontarget -> kill_controller -> fill_drive -> redeploy_app -> restore_verify
00:10:03 cycle 15 done — violations so far: 1
00:10:05 cycle 16: detach_target -> detach_nontarget -> abort_fs_inplace -> kill_controller -> backup -> redeploy_app -> restore_verify
00:29:08 cycle 16 done — violations so far: 1
00:29:10 cycle 17: kill_agent_mid_backup -> backup -> restore_verify -> detach_nontarget -> kill_controller -> redeploy_app -> detach_target
00:44:32 cycle 17 done — violations so far: 1
00:44:34 cycle 18: backup -> detach_target -> detach_nontarget -> redeploy_app -> restore_verify -> kill_controller -> hard_reset_mid_write
01:02:18 cycle 18 done — violations so far: 1
01:02:20 cycle 19: backup -> kill_controller -> detach_nontarget -> reboot_vm -> detach_target -> redeploy_app -> restore_verify
01:17:21 cycle 19 done — violations so far: 1
01:17:23 cycle 20: reboot_guest -> restore_verify -> redeploy_app -> backup -> detach_nontarget -> kill_controller -> detach_target
01:32:08 cycle 20 done — violations so far: 1
01:32:10 cycle 21: backup -> redeploy_app -> restore_verify -> detach_target -> concurrent_backup_restore -> kill_controller -> detach_nontarget
01:46:22 cycle 21 done — violations so far: 1
01:46:24 cycle 22: redeploy_app -> detach_target -> concurrent_backup_detach -> backup -> restore_verify -> kill_controller -> detach_nontarget
02:04:36 cycle 22 done — violations so far: 1
02:04:38 cycle 23: fill_drive -> restore_verify -> detach_nontarget -> kill_controller -> redeploy_app -> detach_target -> backup
02:19:04 cycle 23 done — violations so far: 1
02:19:06 cycle 24: backup -> restore_verify -> detach_target -> kill_controller -> detach_nontarget -> redeploy_app -> abort_fs_inplace
02:38:15 cycle 24 done — violations so far: 1
02:38:17 cycle 25: detach_target -> detach_nontarget -> backup -> redeploy_app -> restore_verify -> kill_agent_mid_backup -> kill_controller
02:53:44 cycle 25 done — violations so far: 1
02:53:46 cycle 26: backup -> kill_controller -> detach_target -> restore_verify -> hard_reset_mid_write -> detach_nontarget -> redeploy_app
03:15:31 !! HARD-RESET VIOLATION: VM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
03:21:02 cycle 26 done — violations so far: 2
03:21:04 cycle 27: backup -> reboot_vm -> detach_nontarget -> redeploy_app -> detach_target -> kill_controller -> restore_verify
03:35:41 cycle 27 done — violations so far: 2
03:35:43 cycle 28: detach_nontarget -> backup -> redeploy_app -> detach_target -> restore_verify -> kill_controller -> reboot_guest
03:49:58 cycle 28 done — violations so far: 2
03:50:00 cycle 29: concurrent_backup_restore -> kill_controller -> backup -> detach_nontarget -> restore_verify -> redeploy_app -> detach_target
04:04:22 cycle 29 done — violations so far: 2
04:04:24 cycle 30: redeploy_app -> concurrent_backup_detach -> kill_controller -> detach_target -> backup -> detach_nontarget -> restore_verify
04:22:35 cycle 30 done — violations so far: 2
04:22:37 cycle 31: backup -> restore_verify -> kill_controller -> fill_drive -> detach_nontarget -> redeploy_app -> detach_target
04:36:55 !! I1-pair VIOLATION: mentes return -> got=None sev=None (want backup_target_restored/info)
04:37:09 cycle 31 done — violations so far: 3
04:37:11 cycle 32: detach_target -> detach_nontarget -> backup -> redeploy_app -> abort_fs_inplace -> restore_verify -> kill_controller
04:39:27 !! I1 VIOLATION: mentes absent -> got=None sev=None (want backup_target_absent/error)
04:41:42 !! I1-pair VIOLATION: mentes return -> got=None sev=None (want backup_target_restored/info)
04:56:18 cycle 32 done — violations so far: 5
04:56:20 cycle 33: kill_controller -> kill_agent_mid_backup -> restore_verify -> redeploy_app -> detach_nontarget -> detach_target -> backup
05:01:29 !! REDEPLOY VIOLATION: grafana redeployed
05:08:18 !! I1 VIOLATION: mentes absent -> got=None sev=None (want backup_target_absent/error)
05:10:34 !! I1-pair VIOLATION: mentes return -> got=None sev=None (want backup_target_restored/info)
05:11:18 cycle 33 done — violations so far: 8
05:11:21 cycle 34: hard_reset_mid_write -> redeploy_app -> restore_verify -> detach_nontarget -> kill_controller -> backup -> detach_target
05:24:56 !! HARD-RESET VIOLATION: VM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
05:38:43 cycle 34 done — violations so far: 9
05:38:45 cycle 35: restore_verify -> redeploy_app -> kill_controller -> detach_nontarget -> detach_target -> backup -> reboot_vm
05:53:47 cycle 35 done — violations so far: 9
05:53:49 cycle 36: detach_nontarget -> redeploy_app -> restore_verify -> kill_controller -> detach_target -> backup -> reboot_guest
06:08:12 cycle 36 done — violations so far: 9
06:08:14 cycle 37: kill_controller -> detach_nontarget -> detach_target -> restore_verify -> concurrent_backup_restore -> redeploy_app -> backup
06:22:34 cycle 37 done — violations so far: 9
06:22:36 cycle 38: redeploy_app -> detach_nontarget -> restore_verify -> detach_target -> concurrent_backup_detach -> kill_controller -> backup
06:40:50 cycle 38 done — violations so far: 9
06:40:52 cycle 39: backup -> kill_controller -> restore_verify -> detach_target -> fill_drive -> detach_nontarget -> redeploy_app
06:55:29 cycle 39 done — violations so far: 9
06:55:29 === finished after 39 cycles, 9 violations ===
06:55:29 cycle 10 HARD-RESET: VM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
06:55:29 cycle 26 HARD-RESET: VM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
06:55:29 cycle 31 I1-pair: mentes return -> got=None sev=None (want backup_target_restored/info)
06:55:29 cycle 32 I1: mentes absent -> got=None sev=None (want backup_target_absent/error)
06:55:29 cycle 32 I1-pair: mentes return -> got=None sev=None (want backup_target_restored/info)
06:55:29 cycle 33 REDEPLOY: grafana redeployed
06:55:29 cycle 33 I1: mentes absent -> got=None sev=None (want backup_target_absent/error)
06:55:29 cycle 33 I1-pair: mentes return -> got=None sev=None (want backup_target_restored/info)
06:55:29 cycle 34 HARD-RESET: VM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
documentation/tests/campaign10-evidence-2026-07-31/state/status.txt
+4 -4
View File
@@ -1,4 +1,4 @@
cycle=1
phase=atom_detach_nontarget
utc=2026-08-01T17:51:15Z
violations=0
cycle=39
phase=finished
utc=2026-08-02T04:55:29Z
violations=9