Data-loss hunt 2026-10-20 C4: register 153 -> 169 (opened R-1175..R-1190: two P1 — shares second copy pruned by another drive's group, box rebuilt after the host delete cannot reach its off-site history; R-1177 proven LIVE on 9202 — a failed local restore destroys live data with no copy); STATUS: the P1 line and decisions D22, D23
gates / gates (push) Successful in 13m20s
gates / gates (push) Successful in 13m20s
This commit is contained in:
@@ -17,7 +17,7 @@ Register: none of the candidates below is in OPEN-ITEMS or CLOSED-ITEMS (searche
|
||||
| D1b an AWAY drive is still chosen as the second-drive target | CONFIRMED-CODE (+ the same unit test: `hdd_3/iskola => hdd_1/...` after hdd_1 was marked disconnected) | defect | `tier2.go:151-180` loops `GetSchedulableStoragePaths()` (`settings.go:1593-1603` excludes only Decommissioned); no Disconnected check. On a real box the away drive's empty mount point is on the guest's root disk |
|
||||
| **K1** reinstall after „delete the old host" loses the route to the old escrow | **CONFIRMED** at the hub store (unit test against production code, `C3-K1-escrow-after-host-delete/`); the controller half (new key minted, history set aside on night one) CONFIRMED-CODE | defect (03 §8a / 07 §7.0 promise the route; D19+R-1154 removed it unrecorded) | hub `store.go:3164-3229` demotes into `host_escrow_superseded` under the OLD host_id and deletes the `hosts` row; `GetEscrowStatusForCustomer` `store.go:3499-3527` joins on `hosts` → nil for the new box; `customer_reset.go:48,155` and `store.go:4189` DO include `host_deletions`. Live proof needs Tester4 off-site ON (B7) |
|
||||
| K2 abandoning one set-aside copy purges every retained package | PLAUSIBLE | defect (breaks `store.go:4176-4179`'s own invariant) | `store.go:4185-4190` customer-wide delete read true |
|
||||
| **R1/D4** a failed LOCAL unit restore: no copy of the live state, app started anyway | CONFIRMED-CODE → live proof running (`C3-R1-local-restore-9202/`) | **design question** for the operator: decision 154 (R-638 A) chose „start anyway"; D20 (R-893 A) gave „put back exactly" to the OFF-SITE path only | `restore_unit.go:428-489` |
|
||||
| **R1/D4** a failed LOCAL unit restore: no copy of the live state, app started anyway | **CONFIRMED LIVE** on 9202 (`C3-R1-local-restore-9202/`): live row destroyed, DB crash-loops, no undo copy, raw English Docker text to the household | **design question** for the operator: decision 154 (R-638 A) chose „start anyway"; D20 (R-893 A) gave „put back exactly" to the OFF-SITE path only | `restore_unit.go:428-489` |
|
||||
| R2 a crash mid-restore starts the app on a mixed state; the undo copy is never read at start | PLAUSIBLE | defect | `appstop_marker.go:307`, `offbox_restore_undo.go:137` |
|
||||
| R3 a cut-off shares restore can be placed and reported restored | CONFIRMED-CODE | defect (R-358 class, fixed for apps, not shares) | `shares_restore.go:131-140` readiness = non-empty dir |
|
||||
| R4 `.fab` export drops a drive folder holding a symlink | PLAUSIBLE | defect | `appexport/fabplan.go:292`, `export.go:714` |
|
||||
@@ -27,7 +27,7 @@ Register: none of the candidates below is in OPEN-ITEMS or CLOSED-ITEMS (searche
|
||||
| R8 `.fab` import drops the family gate | PLAUSIBLE | defect | `cmd/controller/main.go:3517-3530`, `stacks/family_gate.go:173-190` |
|
||||
| R9–R11, R13, R14 | PLAUSIBLE | defect / design (R13) | low |
|
||||
| R12 interrupted whole restore leaves `*.felhom-restoring`, every retry fails | CONFIRMED-CODE | defect | `tier2_whole.go:86-87` O_EXCL; removed only on an in-process error |
|
||||
| **D2** a missing / emptied app folder deletes its second-drive copy within one night | CONFIRMED-CODE | defect-shaped corner of a choice: 07 §8.2 „data legs are NOT guarded" is about a leg LEAVING the class set; this is a declared leg that is MISSING, which the code itself logs as a gap (`tier2_capture.go:55-61`) and then reconciles away (`tier2.go:454`) | P1 for optional legs (no other tier holds them) |
|
||||
| **D2** a missing / emptied app folder deletes its second-drive copy within one night | **CONFIRMED** for a MISSING folder (unit test, production code + real copy seam: `C3-D2-missing-leg/` — mirror of `photos` deleted, run „ok", the warning says only „nem kerültek a másodlagos mentésbe"); the EMPTIED case CONFIRMED-CODE (`rsyncMirror` = `rsync -a --delete`, `tier2.go:744-751`; the test seam does not delete) | defect-shaped corner of a choice: 07 §8.2 „data legs are NOT guarded" is about a leg LEAVING the class set; this is a declared leg that is MISSING, which the code itself logs as a gap (`tier2_capture.go:55-61`) and then reconciles away (`tier2.go:454`) | P1 for optional legs (no other tier holds them) |
|
||||
| D3 / T11 a host clock ahead silences the whole-guest tier; the hub reads the same clock | PLAUSIBLE (B6 #8 on the rehearsal VM is its live test) | defect | agent `localapi/server.go:1178-1190` |
|
||||
| D5 a dump that fails its own check has already replaced the good one | CONFIRMED-CODE | defect for the ORDER | `appbackup/dbdump.go:388-422`: size check, rename (`:399`), then `ValidateDump` |
|
||||
| D6 kept-data page can list a running app's folder | PLAUSIBLE | defect | needs an unreadable app.yaml |
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
# go test at controller 7ca59e2 (v0.336.0), scratch clone; the real copyTree mirror seam (rsync --delete stands in by copyTree for the emptied case — see note); 2026-10-20T10:19:17Z
|
||||
=== RUN TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror
|
||||
zz_rehearsal_d2_test.go:22: POSITIVE CONTROL after run 1: mirror photo=true doc=true
|
||||
zz_rehearsal_d2_test.go:33: run 2 err=<nil>; status="ok" warning="Figyelmeztetés: a(z) app alkalmazás egyes adatmappái nem kerültek a másodlagos mentésbe: appdata/app/photos."
|
||||
zz_rehearsal_d2_test.go:34: FAULT after run 2: mirror photo=false doc=true (the second-drive copy is the household's only local copy of these files)
|
||||
--- PASS: TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror (0.02s)
|
||||
PASS
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.030s
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
package backup
|
||||
|
||||
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate D2). Never committed.
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror(t *testing.T) {
|
||||
m, src, target, prov := newTier2V2(t, "app")
|
||||
prov.has["app"] = true
|
||||
prov.binds["app"] = []ClassifiedBind{mHDD("appdata/app/photos"), mHDD("appdata/app/docs")}
|
||||
mustWrite(t, filepath.Join(src, "appdata", "app", "photos", "family.jpg"), "PHOTO")
|
||||
mustWrite(t, filepath.Join(src, "appdata", "app", "docs", "contract.pdf"), "PDF")
|
||||
if err := m.RunTier2("app"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
destBase := filepath.Join(target, "backups", "secondary", "app")
|
||||
photo := filepath.Join(destBase, "hdd", "appdata", "app", "photos", "family.jpg")
|
||||
doc := filepath.Join(destBase, "hdd", "appdata", "app", "docs", "contract.pdf")
|
||||
t.Logf("POSITIVE CONTROL after run 1: mirror photo=%v doc=%v", exists(photo), exists(doc))
|
||||
|
||||
// The household (or an app bug, or fsck → lost+found) loses the live photos folder; the docs folder is emptied.
|
||||
if err := os.RemoveAll(filepath.Join(src, "appdata", "app", "photos")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Remove(filepath.Join(src, "appdata", "app", "docs", "contract.pdf")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
err := m.RunTier2("app")
|
||||
cd := m.settings.GetCrossDriveConfig("app")
|
||||
t.Logf("run 2 err=%v; status=%q warning=%q", err, cd.LastStatus, cd.LastWarning)
|
||||
t.Logf("FAULT after run 2: mirror photo=%v doc=%v (the second-drive copy is the household's only local copy of these files)", exists(photo), exists(doc))
|
||||
}
|
||||
+306
@@ -0,0 +1,306 @@
|
||||
Tue Oct 20 10:09:14 UTC 2026
|
||||
filebrowser Up About an hour (healthy)
|
||||
felhom-controller Up About an hour (healthy)
|
||||
paperless-webserver Up 3 hours (healthy)
|
||||
paperless-postgres Up 3 hours (healthy)
|
||||
paperless-redis Up 3 hours (healthy)
|
||||
immich-server Up 3 hours (healthy)
|
||||
immich-postgres Up 3 hours (healthy)
|
||||
immich-machine-learning Up 3 hours (healthy)
|
||||
immich-redis Up 3 hours (healthy)
|
||||
traefik Up 4 days
|
||||
0a4fb1a71c72cf704d1afd44e1ab24a8a68e11b8d82ef1dcb71a75d61a9dc1a3
|
||||
0ce36c2e6db54e085f71a1436c4c89a9807a930c373198f451fa1665f90f0cab
|
||||
0e2ca5e8738b95134285a666b1a7622bfcabc4728fb9269cc972dfaccbd59d45
|
||||
0f4002ab75695fb312775a8203cc5213b256f1b01d255210cb5cdd0e50cd9070
|
||||
1a8fe48409be3defe01713e2f621091630b965e1805cbb5872d1738043efd35c
|
||||
1afe3420067df65e0d3e53baaec3debc39d2d42d333c7a3c266e4f6ff98e04f7
|
||||
1b41940af5c4fa82af59320d6c81ba622d97f2e43d8ec51c50dcf65c2fdcb1e8
|
||||
1f5120663962c7b03539bb993ab736be4800096e3d1f7147eb922d44b308ae6a
|
||||
01feef94eb7e38a3679921b66cfe206e08adfd9b360ed4a60414cde606de1643
|
||||
2a71e17dfa1e930ffe0c213cd1615c8d1adcbbf8d61615a38064d331c265c02e
|
||||
2aaab84bf843817d9e6f32b673277fd1e0c2c8c2d038b8126df44927b1b41006
|
||||
2ac49fc9cc2839f55c8abae76039950d15a5c301ea5b3918fa3e5d8715a15f42
|
||||
2bbdb463b29493d60d352bafefccbe57856cf80bd2cc193b61f9bcc1310fb45e
|
||||
2bc95b104191ffd31bdfdbe1e038d284d0624c04735094494c2c77d957e8e905
|
||||
2dcb4bd2c3c618d71f582d33d5b1243e6ab075d3b00abc238d4430668a2b6f55
|
||||
2e4c79db25c47436b44245376b1f6ff1991208ba081fddce1938c3897fc63c07
|
||||
2ef8994a563ae88024ba29db4e349d26e6e3889e0683d30d1fb258208c9a10b4
|
||||
2f08e882c5d23373e2df5486dce28d4728330f9f0131a2735ebc938bfb4d7b3a
|
||||
02f42b55ee403ceef63a472c3624da0b48fab9e6aa6dc0c7800ec68b2f2041ef
|
||||
3a3d7bc33a1ba21ae4a39c8cb345254cf66cd1dc60afae64e2299513b094b93b
|
||||
3ac9820f84d0a50f233c63a62ee1b63b92acac026d68c60a8227f370b0be00b6
|
||||
3d9a93769342e83a3c512d517b99cd64f8e794eb305c009e25c73a6f6627ad29
|
||||
3d9c1f23abbcef771a4c38a75d75865a12d76a4359fe83c1828652dba3f91710
|
||||
3f015f020f49d4967d2ac71b94060e6eaec86bc8852cf9a0b5f3dd8133dd528c
|
||||
3fa030697dbaffb6809991bb69a335dc4e9accfafd4ed5ee953ed676673912e6
|
||||
4a9ed2aed1fe75a0b02f688e29a20acc2bf7e081ba7518a7d75451be30fdc4e1
|
||||
4bbaae5ae8cc8cc3330a73bb872b16e1420805197fa9bad903f7db9a1f358041
|
||||
4c54a4cda3c5280d5b13facf3089f8a0710c4570876aaeb679025aabf92df99f
|
||||
4e052b4ef5602a68eddcea049bbc4e79b306012cf8b7f0c9cfc3e6c39af64fce
|
||||
4ea780981a54c3e87aa72eb80436e02f3c6b3aaeab96ecf869602f5f5b0b546e
|
||||
4f2ebb779645de5cf9d24e93c6fbd9d95d358b390dec94eed6f0223016785e42
|
||||
4f2266bb0b8a9b902eaa439ae9225963d6fe4359fe30569c2d8afa926b582585
|
||||
004ae946f5997471f4554ab7158a3b982b72391b59c59be40ea79b501eb1dab9
|
||||
5b84763ca63a28ae79c1393d700c534e7dd6e610f890f540981e49972e3854e9
|
||||
5c7dd57f445d87a39eea1e7b299e1a0acbbc946d6bfc92ba165c4af8933b322a
|
||||
5d47a8950cfd56492e0f8b4ed6bfbf470a2e22811206ac0da0b79e520890b41e
|
||||
5def44013bf255d4cd97c3829ac7ae895c607333a3ee586a4090bda128d379fd
|
||||
5ec44fe3223976727132521b3055187e88783716a1f5fd07d1625205acff9888
|
||||
6a84846fd8e476d206eafb8f5a529ecbdf6b1021daab890429077e32ef4c2b6b
|
||||
6aef4d0808bef53a15fc6d19614b641a5e36c941f2729572cfef8cfc8e36bbfd
|
||||
6b5a55f08cc63514e70a93a3eb5d294e9b109e990521bb0ddcea4818a8df1456
|
||||
6b8acdd640ae067c2ee5235ddea8ad4c0e1eec2aaef1277b426923f56303e07d
|
||||
6bcaa587af8bf282f83605713213c20275b1c8178c28de97b154af25bb6a674f
|
||||
6bdf1f89b902e5a7134a87e0269229531ec6c8391256ea1bf466d2d6b66dd8c9
|
||||
6bf7012e27730a047bff7d69ceadef9c06433783d0b9582faefa9c9d0f966c86
|
||||
6ce965b12082d21ed8ade00884de6477faea9f97cf7a85e4b5ab3302d3517fce
|
||||
6ddb2a450b19b940661c677b20936e92e8de54b8bc76f8dbacc56398d7d13a38
|
||||
6e21c6398b13ebac045a11b136fbb2d289e6abe99401eed9108b5464754a4c87
|
||||
6e0521e4314af444375b56e348a5e3d34aceb88d2d5e2f48b6986c27250e395e
|
||||
6f19aa563c4e3e2000ed997157bc7f5bc8a45ee1d77c2cdcd1fd239117a62705
|
||||
7b5530f7f440ff6477f2ee5885b37ff9cfbf933d7908225998875b233b4d2a2a
|
||||
7bfe063b0155297f5bca22c7ed02ae7ef9ce1b968670607415f7720e1e8649f0
|
||||
7ef24d6bf9fb190db2572505b489e45731a326c5c9c4ae6e36ffd7874560bbab
|
||||
7fc41ddc47424ea11d4940f7062cdae69b92af645e6759f8759f47b689a19cb7
|
||||
8b44a3824395c930d0db784e5650ba7b58663090c3bd72a44f9dc716008a9642
|
||||
8bdcbc26d645ae3bc60742debe32650510bcfb7c69ad4416688af5b1815e5645
|
||||
8c2a7cf3f809218d20576368d4d419a63a07a89c3b12cc389fb2f3424fb45b73
|
||||
8c4d45c90f336c0ffd5d619765d8d2a45e802d619548f187bb3f2672b3534969
|
||||
8cae3c834dd9284d1d0069de042c2d50e49a138a78d93a59eb752272e4a5e4f4
|
||||
8d2ab3e77a72ebd708630e22f2eefc0a66f73cd94ed2abbfc4f666c504a3f172
|
||||
008dfd2a2e98c6b6673ee471d72b9e150cfe02859eb3a3bbd31a3c582b5a3a46
|
||||
9a2e6958ba1ea4443ade7619d326c84c50dbc444810b47904ff625969725a091
|
||||
9a86729a66d00d8be95f93062381b9bd91b16f94b654636681553fec6c7c7985
|
||||
9abdd18e6a7b07800067237c93d48448d49d9ad2388a91d35653323c2bed4389
|
||||
9c59fdcefb7ed258e615f8cb688334a69aebdb8e38fd1075a60d6fd98167065a
|
||||
9c033896f0b00023816a49cea882a342c9b62ca4528640969a0d22c21f507505
|
||||
9db05859a329f9544f5a48cbd987f02fd00e48f160dcf6ee8331d8accfb423b4
|
||||
9e9fa98cb2344c5eda9bae7417557f2bc2503de1eed8219fdc2d2d378e1883fe
|
||||
9e06437718bd8f9a27240eff7b425406aae50f061ea47a6991b9830ee2af9f59
|
||||
9ee6f261789b97ea0887a6a905bf5bba61a7070cc91ff64cda0091923d9cdf25
|
||||
09ea67268ce8c0442357ba316f6431cd96143da6bb54008d3c342be8b6ddb169
|
||||
13b744ec13f00bd3e5e2ecaa37bf5385af38cf0ff94fd05e660cfed3339a0c05
|
||||
13d76d014354ffb4e5aabfe00eeed5206b98c502dcfd515545fdc7210316bf61
|
||||
18e51d43a77ac282e108fd8a8e55207282cdabd2219af1e47c2c214117179f51
|
||||
20ee459190ea3442ab7cf0c288182033fb84b75a15bdfc14540e80ab60dec542
|
||||
22ab14c474cdb6cf9f873dcb6c8ce85dfc588abb46e2bd0925cdeff384437c46
|
||||
24a30fa240cec28f9a8b6d02a47ad1fda39674f094ae095b0d6ec3cbf099ed44
|
||||
24da0da42dc2eb9b6d35dcb7cd513b1e5c7d2ee38b6d59d9a1d51f0d01f6f920
|
||||
27a56be2953f040882ecd07fd8b91937f8cb17c25d1ceed56656ed2075518f86
|
||||
31f3b426743b1cff7952948995915586480486ed7fd06eef3379cf54b8fb3d9a
|
||||
38ba44b0ff38f8df925512a477b8b5b4cba216a8ec0bd51b7463a86a88c42fea
|
||||
43d6928bc4bb7a1e097122424e35f22393a0ccf2e03863d7dc3cce44163c8ae8
|
||||
49d0cebfc3199cef395ae27a6ffa479340c1c2f69da9248a777d7bff9b6b9ca9
|
||||
49dbe15539d7170822958bb0699d140c4d493215236b32679340588710d5452b
|
||||
054cfa9b87316799ecd24ecfb480a38aa04b5f85a478a3db95c41706e5b63523
|
||||
57e9dc8d409d939d877a513023c95146400dd10078919ab9402d8144d5f30ca0
|
||||
62e86d5d6aded4a8ba536696c0df0f0af89f2cd7b4b893a7383e3abc00eeadc5
|
||||
63ddb5c60e6cf39dea412903cf1d6c0164a1146bcf4ab98ee94c9b1f9e45c6bb
|
||||
63e2655474549ef03aaf7d9d741502287d2f01170c5ca008d7b09b8e00b1be66
|
||||
67eff4c5c60d493dc2a9f7841f7f138b610dc389c32793a27a62ddc13d417a1c
|
||||
68d18a8d3519b6ea6928bbca40f9db7500b700f66e6eab236e7c32a214de2cc7
|
||||
68f83c9613471fe0e046a1f9252b71e86aa50d36315c675e92dc44a7cdd0267e
|
||||
71cb6156444948894aec343c9a2c0c7674aca84d292d090c7f786d13146709ed
|
||||
72ed18f5962e48dd5fb54e7d91d2fd0479017c27b2c60fc58a5d6b6e55242757
|
||||
74c6b39b7b926dd56db16f175026c2bf9e80a912916f273c122227672f181d03
|
||||
075b39bc7ae8c56a6d99923e4938ddbbb4174df6f0e653abf9c99b420b401a31
|
||||
78c01e39d1997eb54344f02f14dc02c7230cfb14ba58abc1a11feab2e9af85be
|
||||
78e7acb9228c1df4ca4c706d459b97c8e79e8399fe06476ce72bd919a3b2b77f
|
||||
83a8287d1e8700aa7e207b2c617a68557ce6f83f373f2c3c6d1443ae50032826
|
||||
83bc388f3ae110e72ba51a8f0b68f7bbdec3685907c1ff08095b63a1759502af
|
||||
85d6f7309436e59cce9e373b0aa1626fa5e12d8d81163773f5f2c45da9d56543
|
||||
88c6573974e81f1fff96999bf54b80c66bc6f1e9af6746b3cd6801616198e0fd
|
||||
92e2e73e3aef6e5330d2ce8aed3c58e7e3632a6f191f2b03c997ccabd245bb65
|
||||
93c2b9d0a1f54c2e6355d206d93bf2309f81076628aae9d868406c288b0f6bf4
|
||||
98a80f5f7903ef45f92f74a91f34dc01d527f8a2fe98b4fbe76711f5259416a5
|
||||
99f7aaac2646c66dd0eeeebc157b8563b56299c27ab4bf64e13920d089ca7156
|
||||
104e16e4796c52fe212c04b06417161e0bab20fc01632bfcf71421b73b979fbc
|
||||
122fe66bacc9f04b3f3e4f6d0a3b15abdef8321eed36637a1284e3bd145463b8
|
||||
158c590779b033e48b5efcf895fd0bad4c6f277303d72b963cd0802ecac1f440
|
||||
218a186ab13a38d514cf9f523e5798452ae4e5ec76ecf26fe1a46ae22b47a508
|
||||
283e9d53742a13c471e5f59f53a57c179d461f0017ad5f8db27881cdd994cd04
|
||||
305b75817cf5c06105b19dc8c56ff0d796dc6e267ae5bcfc6018aa3d12037744
|
||||
325ec2aa2a6c307bb82d4dcb600fa3ff127aa6d1f35dc683ab61124ef5f0f5b3
|
||||
338f7535ae9ef91754e76604238ca7652273ffce9e6f0a1300de0b1edb61da77
|
||||
343d41a3b38b5a7eac10750eb0213b81780edd3075e9a1fdf3a2c479f4d2474c
|
||||
346eaddf98e91d6d6e7b8c526074c27847e4a18be01b41027bec93abc2b7b2fb
|
||||
355ba69bab94a95e00e87c6c4808587881b9675699083e087917ec8492bebc65
|
||||
426ccc08685def1c9025c94d343de0e59713073984634ee226e69634d6a18fb8
|
||||
552a4b340eb6be03c188cb17628aee0f916b544142465f8e52e7e9d2325206ab
|
||||
556c805068afd3f6784bda2eb59642d7dc03756580e959e1c75262711a30d4d9
|
||||
0642eb9eee222e235de9310b9bd86b42493e5f4d71d430619b5832484dd09cd6
|
||||
645d70ab53013564fd243eae61db209e3793b826a9eeffd77395b18e27458789
|
||||
670e6447721f667df5222cb06a68c2682eb92803184a44faead8aea0dff44df6
|
||||
0717b5f7615c345fe47ab8212c06f506e08585551d93c6a8083b56a216adb71b
|
||||
734e6b63f955d0b508dea39a8637755e75fbb353d54430956a0010f607eb727d
|
||||
748a1560ef6216c8bf80545dbeadd29f99fd5c984b6f34bb678febf8b369adcc
|
||||
748cce6aa136bdc0b3111c73a7c9434508200f8c231933b925d42b37b9cca17c
|
||||
760cd2f9203dc3dc3f933e4a34bbaae857dee0afed66e727846a4c7f750ab5a8
|
||||
762e15e05b2f84f035a7011e10c64558dc6637c9fdde1846dc77f4056f3962d2
|
||||
787dc06078f3718da6faaba82269a7d21b213482c9e45025315bdb6cf59010b5
|
||||
866fc46ac37407bbf74761cecd21fa6322c4e7621f3dfbe963a85671d5753e5c
|
||||
921d92a1c692ea0fd42cbe500bbdb1f3defcd5fb9b51df216451aa260d95ad45
|
||||
932bae2149e79ef113eb4929614a895c642960160d5f5cb2f2e9afac7dfa6af2
|
||||
968a5d7a54fb1d8ef4163f287e128cbd541ab46a2cc54db721dd3eafe1397b65
|
||||
991a5d344112325fb2c40700cd0ee96a784ce2150cba0a7b0be3ed6eb2411878
|
||||
2469a8c88a452ed226dd6d1082f0cbfbe3107d76f919b930b6bc7ce3b839ed90
|
||||
2634a60c3c37d5eba4be4827c08c0e99a951b934604c29d116fcddc27808e480
|
||||
2788da1d8f917a9d5b8acab2c7fe7336b6a7810f1c7edab63ed327a6022ceddf
|
||||
3000c7863b2e80a72329ca1fa0dd03d2fd3994a2391e82450a8c92199cbb56b6
|
||||
3461d48cd9e88caf5363b0e4d96e799f6759ab8b8bc6ba4c148492a1637c01f0
|
||||
4681b8c67ad20fdd3ad04db895084d37a9296ac76b74ea7bc9465bb3ac93df8f
|
||||
4690cfe7c61625190732c028ad4b918930a65e295005171d48114aa8f27b567b
|
||||
5206b60d49110c01e05dc24034e12b767efa739e2ad21c252ab83db47e56d526
|
||||
5234fcace55ee7e44631499ed1f95fd0289918a84bd9f5cf6d9f761d43c34f17
|
||||
07402fbe22b572302f3317958f8cc4c6ca30317e0ee5bbbe9ff51c60b3da5a05
|
||||
7458b4c4b975debbd2bb705e9575d2d1ff5e9308ed9e70c302f5643136cd0cc0
|
||||
8097f97f4ae00b654b1c0b4408ad585f5bfce7999e482f244bd698268e1f6061
|
||||
8158fe1f757d26cadb484ed9690ad4d20d4cb382eeec7c9b0627b365afd6576f
|
||||
9917d608ac0ad05856d31873432d641828ef519825a3f451e6a1ab2519dbf0d1
|
||||
13175cc97740c834e2404c11ba9fa9418d0c390d27a87e6e869977d6a86c20c0
|
||||
36571aa758bc335acc463c3ea074422a5123c309a37e1a7fc818a2069941c493
|
||||
59438a5f6378d218a73d3106ea1984d1c160fe4fcbbb7ec1177067f17430dcb3
|
||||
60496cd3a1e1ad1dcaed30427b70a57b8b14a122bae7f2cd882f8154a4549978
|
||||
61544bee8e2c1cd65e999121bb63011a9c1b4139b20b04fa32e3a850ca8c82b7
|
||||
84046f48a922cd337284bc2448a5d2682ea6a2e3479e71531122aff9ee4cbc93
|
||||
86022b0cd1043facf7b1c933c3b6a6ff36277a76d50bdb2da6a5690d57381b9e
|
||||
98129d711a1e807ab7c3ae4c97187ab8505cebf1012218e8530d0829a2ad436f
|
||||
177451abc7ed0606468250a7793e018725e8c7f370ee8937d4941a11571dcb9d
|
||||
0217869eb1be5181b7053b993443dac73d9cd0804cd5bedb604edeb43f911702
|
||||
219118eb01fa7be03fcce2124b74708cacafe66aea617fd2bafa32f0b2d4fd80
|
||||
394985d0c3db0791b8c7569fccabd36959d00deb285075f9852568c4c81d32eb
|
||||
722380f945884d4dea921ffb958513e02a4802483ec0886712e5ed3fa28d751c
|
||||
774878c58b6a8cae6fbecdf656f2c1f09f37a89190068663e923763dbd6969ce
|
||||
925226a74657c7ed0fcdbcfa9b504a47f7e21d0982c44f7ed9576140fb417146
|
||||
963079a9efb903a575b363b27e15927312569528b9066511b4c282e01e1ec931
|
||||
3830913f0478ec59a7a31d538f858f8ef612e1b2ce3bb9058a2d18f7b43abe9c
|
||||
8579968bcd362a1aa9dcf14e22956b2fd15fd74a2fc2e7369ffb1457c2ce6693
|
||||
8678479fa55ea3871598eb6914eb8428ba599da4c619c62f231e6931f0cb1f54
|
||||
13694424a3895e8e73cc1e06af29aae0c3179c3cb5587b9daadeadcdb29b7cc7
|
||||
67711784cb23efbc0762e83f03365b837ec5216c38f3ba682a82eaff6a7863db
|
||||
92236992ea73ecd9befe3330e88e96a7aedacd9cbca0bb8c32c31d0e2c914e5b
|
||||
134995217d29dc47fb0ca1a274ceb5ea74225d36a98c4bbe6e07bf8c8d6898c7
|
||||
481706314b835ab042583c3107e168932844a162a2adf3b9ef2fa27433a7fd1d
|
||||
4098005900d38c71b152787ce4bdedfb8e2b2b8b061f87645b2b4a14e0004335
|
||||
4699188665aa6eb965b7bb9f4938187f377e7a4ad55fc97e44802ea76f3612e9
|
||||
04762469942ae24c353f929fced777f245bb9f2f888a9f2044ff27fc45933665
|
||||
a6c297b9282b39caca7e513724219af096a3cf6fd453c3736ee7c4c833ef417d
|
||||
a22cb584ba3444bc1dbd90b3723b1ad9424352251f2218a92b1884ef74e63b9f
|
||||
a68fad1bcb1f1030af3811590cc18d74eb1da3c67190af4459fa69118aa8f035
|
||||
a74b9b2333ee936c17edd0525b49fec1337768ccab68f5576a39486c875ff968
|
||||
a74b41238ed9ded9e866679ab00f85813d800f92e4bb2642d3f9de28a1655b81
|
||||
a84ff849188b2bab211fefe8bc61a70f58f48f6a2a6ca25c2645e0bb5c384cbf
|
||||
a2815ec8af59e0a137aa0cdf2baa48ca6d1b86e291c5728fab9449280d15fa75
|
||||
aa5cf2d08c68b8d3dfe3330861935fd97ff8fb9169645f6c632d970ad71b575c
|
||||
aadb8f0be78890f1e08630b7a762a82853dc1402326bd16cd2a2844fa63fef1d
|
||||
ac48d92438e8d19cf546764cc051172af82092223daad8ccfbfb4cb05100f5d6
|
||||
ac78e0a5ac10c7d0239e0e1ba3b604cd6fd418d87a6f32b66f3ce98c674164d8
|
||||
ad79ebffc7d8225cfbee6ad5fec4092ba7376f38dca912d6df8ea932f1eba20b
|
||||
ad7084f002b17a1a27f70c796e3d7424a064b2ef5b71eadb7b3b08dcc9f59042
|
||||
ae2a94eac2960e95f944c9a4a0649c94c405fb0e518adfba2e6fa0f80f2a5af2
|
||||
ae9d5c15cba1adf881044a4449eaf72f5f2ed753c301edc1e43bcfa4560e8d31
|
||||
ae5499adb73000b6f460e63d6e045fe492702bd73187f868d9988690cb72a453
|
||||
b0e341cb8fb7f2fb42f7a8512a9b922591cfa40bd3522f31a422a9fdb4586008
|
||||
b4bc9ff2d9b533e896dfc5082ba865a4ea7676d71259d73aa33d515a5a60dd91
|
||||
b8d2fbe7e6afc931581bf5f1edacaefb9071bcc9f709e294a3f5ead104f73890
|
||||
b8dd8697ad52f98dc44ed3c940752aa52e16d6ab7563d36b1c9e221db22d1e11
|
||||
b21ec6b73a9f1f51623355f2bf8229b5f6f9da1484e1ce105cf8e0363475a5b8
|
||||
b081c3329ed821e22b0ee9cecbc2f52a5e9be70be2b1cb83bb262c21c84640d1
|
||||
b274ece5f6e1d506ca0646077964797c3e7b4413c9a9f1e453d9257e6b4d09f9
|
||||
b676fcf4f5272015b942c8589135a18fa16204c5aec457f1d0898ce6c84638eb
|
||||
b1591e2a2604b6404d0d18e4b2b7599aafbcfb2bbacba06b5de5ff14abf14be3
|
||||
b7000ea518cd327f7dfb42858434fdb80c451aecf0a0325b6ccbff1f30545803
|
||||
beb17a9d330ce361923600aac8e13f8b10e62d0b4067380ad7a95c4d77f0110e
|
||||
bfb91f5666855fd57228801ed0a4294981bd2a6665a93d1093bf3c4a0e89cb2c
|
||||
c0fb7358aabef2c38e6921e2ed06b7fdddddecd29d530814545968323eba964e
|
||||
c1d4fc53203c4b65abb1ff76f9505a46175c57afcd9c95cf445c6a3c443839fb
|
||||
c3fbcb924a928bca859b69c75e4df275eb5e0abf03029785247313b0fa1cdd27
|
||||
c4bccf1c99f4990b0a23848b6fa58977dea0bdbf6ab69b7c0c2b90f7f0a2f5c3
|
||||
c5e875c34f4e4f1fd4edb1c6dbf4862a6543e26387ee480eeba0f5e3f44af7b6
|
||||
c8bb03639a4d7c41c233fa28d5a1c06dee874fe00614f764df9db3df14da86b5
|
||||
c16aa92f949bb89a034c6ef22ea8ece859883ab309086a204e43cedced8dd059
|
||||
c31f2070f92b36708a62c1d53a7aea495fd997440f2c01c851ad83c66357e379
|
||||
c35e0b961694a882dcc4c7bdb8e657d4b0685687cb5c4469559ce5078c95302a
|
||||
c56e78b468dfe3991fc787ab7fe30b4282a73d7fd2e79e4dfc9305bb064685ea
|
||||
c67a888239bc5e11ed44ca31307f3b46f7f76698d706ffc764024110d1a69ceb
|
||||
c424d546895a59e0f88e2296a665c03f4618ac304ef19742b6f1c543665b9bb3
|
||||
c716ebc8ba8fa1556e1dddfaae98c7019618d39ece0e78dbd1b9f58dd0760d94
|
||||
c797d99e68b0efce9b1337dc0e3e9cbf42806bef03021ade362cb4e4d1de4222
|
||||
c5708e0a13ababf73203b1e297b3b73229cd6370e3fec1653b7cf1dc143a0d0a
|
||||
c52417cfa3d923ce7168b7c2b082eb3ab31bc1750ff3f3edfc2c1cd59fb74f3d
|
||||
c53655b36c7abc28db3020511a0ebdffe8fa37d86412753fca86f4caa7ae7a6b
|
||||
c782734d8be184e0721d0a984b6ac3e8e0052206a9e46d22188a7fb2f923face
|
||||
cb71ab036375acb25c69afd915261acf65f919008fbd59417989f6ebef110c7a
|
||||
cb613ceec1bf86b718059ef3e864dce7ed60e5b6a7bd5ab9f7c633350df48f3a
|
||||
cb10732ef556d77ec0a0a5d7b29c0789b47a9757b67bdb3fa9d5be22809debd0
|
||||
cf799a4b43f87d63eba6042a73b9eba3f17b354be1fd2d94dfac2048e13f3fce
|
||||
d3c941cd28b5a692c2532f1381afdc57740d603fbba5eddbe3539f561f26a0ca
|
||||
d5bd7d95fbb1f56d7617d96a5a6deb65417b4448692cb45f4a0bedd824165fc1
|
||||
d6a74fc39944848c79012bb06d915c609b92c1099881b90b4cda59074bd1124f
|
||||
d58b42f643175b6272bb579d354a9510bc34fc88e4e70dda08346997ba3cec13
|
||||
d256b9490d623a136093e459b9f466dedadaa0d00235d6934044e6591ee7e2c5
|
||||
d352bb11678cf6cb47a16740ee9b79cfbf4c19d4da23f4fb95621bbdea6b91b2
|
||||
d053441be51f569ac393c26c220b6369ae3455ae76f3627eba223e8b4e40eacd
|
||||
daa717bf6363a3afe8dbcd3ab863a81ca737e0999242bd99c618daa0e4fa085b
|
||||
db0c37c3ce9df2f2090c622eb4194e6860e26846398d4b4a092b9c4c21441af4
|
||||
db2434b8bc6fb7858aa5eea51b23eb3fe17ed8e8ac025596ebb4f857d00f9ba9
|
||||
db6910df9d6429a8ecffced74f22c26f45a11d48efea51e48e8097bf43331540
|
||||
dbcfeebe1162c457fdf6f7cbfff66c971dfaca8a1a8ae2fb5472fb9007f83ddc
|
||||
dc5c95c9a8f2f01a8eab63c7d591a7d008dbaf5f4b6157380faeeaeab168cd59
|
||||
de81766fb8e806f93c97771ae1a0ff529831bc5e7f50e2187d41dd606b8e23f1
|
||||
e0e25d3ed1797b0092c897ad904a4ac7a2bd77e12b96e1790a939aa009851803
|
||||
e0ee75c206f1b9be41e608c6de05f8fad8532e3bda240cfad0640c673ae46395
|
||||
e1ae949c5339eca43334dd6720b49df19082036b8f53151b8dfac97f2d2c0b6d
|
||||
e2ca0579185aa6d7324f144357323b4d5521c1e94e8dc7929f0ffa9a73e1e4cd
|
||||
e5f9a9a8f7e4b6d097aef1b0493180a547dbf19006920433e7b80aa8aac0026b
|
||||
e6a40ba40f9ac5f43ba2276b3cb6110bb4c0e20003e866a55ca28c954ddd0e9a
|
||||
e9e1b0601616187e8934c7212e2fde5b590b50308aa9c8c5dd12e419c6bebc18
|
||||
e45c71e0680d2856e2945881fd01ad6320868e0a36427c6bca637c53dfecc625
|
||||
e62af03dd8d599dfbdbecb8c6516d95e582ee803f2ddd3798479fbc769229030
|
||||
e72bfb25201770edb1889f678f5a844a57c9f5308acd57a86c5f4936b800d441
|
||||
e311fa91eca24a97e0c558e91d06dca353aa9348a7ea4b111e694e181868fc27
|
||||
e339a495e0c048e30483e4fd3bef66186712975f3c33c90ef63a3c9686da3160
|
||||
e668b1c1f44dae28ca34f530e313c1e544751b63bf0c480300be8e242683fecb
|
||||
e4638cbb5b12548a76a2cb6e87c7f2ada75d61d3c7d1062535ef0bf2b3cde05d
|
||||
e7945baeafe0ac6d9a1e6761786c35fddad76c81d86e544d369188c186a0389b
|
||||
ea35c704202ee36415d4f7a0873bcb254d30268450d7c9ed2dc611c1e2f4d5d1
|
||||
eaeb108b23d455d01ec0c9e44d39ff395522942794d26f3c149e29615be9bf1f
|
||||
eb420ae07f9c459abc101c1264fa8cf779002ef20748ef97abe7c3764c24f032
|
||||
ec30ca07d0b0af16afe60f217dedf15573045ed9adc45e21c3592ee6716fbe39
|
||||
ed02b9c56e48d34910bc97ebff834f1cb082c23ebcae97b0f51f3a2633664466
|
||||
edcc77bb3d279e3dfe7d687fe30e22f95f5d9391b9908e61b2206a6cf309ea91
|
||||
eea963e8499939d36d09e42d8d5cd86798e81e264c4d4c012191e42926e508b8
|
||||
f1b01ee5efb2117033ddcad5f5eafe8e6f65ad6fc3e933902bfc494c13a7d3c0
|
||||
f01b151ee48f9a3a631457069ece3a2ac3e111df28110f3c06f221ee7c54eff8
|
||||
f2ec30f9f836ed26cb13fdb80064d1dcabf953d1952fea2d85bb6410f4cb2271
|
||||
f8a87cab6bc0890c7a2e36f73e533eac1a1cabb13b886b8ed71afa4c8b7d90be
|
||||
f9b0fe025aedb49e94d80119a8a8dc0f85800bc592e568dc95b0f1e5439c54fc
|
||||
f56a1ab986e83e118ff174e9805f6151e47ff693c17a2f475fcedd83cb98e051
|
||||
f82a956615af08264b64e051e09c1441152d92371d4d2db8d5c0ac4ab47a7182
|
||||
f287b03a768126d72af210dceab12e329573779e7aad56c123ac0dbb14fc4f4e
|
||||
f347e779fb8c09daba2a757dfc3ee7c415f8f5a2eaace19535d363cc3ab3171a
|
||||
f547c232f5e1785aa11200ac03dcfda6f056400bf1456a3388db6d73a8158ca0
|
||||
f6822a239614fc82462535db5b2806dd616696c9552e1d469922f4b7cd2bf56e
|
||||
fa453f6fb451a564d31e704bbca0bcbcc9ad1109a25a8e5ad5c704789a424c4b
|
||||
fb5ad60698efb6ee9f45fd440e56c060ba01ba03a854f556e42f8090b2057c7e
|
||||
fbc407de7b12a4975b60dac518ac287f076659213cce708871162342419ab383
|
||||
fbf633eb758b7b66ab761c34eeebc759791e2f73020e839b3cfc2d692c9ba21f
|
||||
fc1bddcbef8c02474db71bf11dace70e1e51d94f3c3fe6d9d45f4a669ac4a9fd
|
||||
fd453dd47dc82ff7625bd4093c62140fe14a348814f094b43b519f7f59293af7
|
||||
fdc05b3bfda3101bec7d66b5e0cfb589e06f1f4b4d6b2f9d84dac470b0abf53b
|
||||
fdebde6f8bcdd5281c50bcb4669c45dc54e33563bec872f8d23d76d2fe3ebfdf
|
||||
fe503d3e9a34b113d4c56b59c5986e13c44b6e401b6e6681eb62712d24368329
|
||||
felhom-controller-data
|
||||
filebrowser_filebrowser_data
|
||||
immich_immich_ml_cache
|
||||
immich_immich_postgres_data
|
||||
immich_immich_redis_data
|
||||
paperless-ngx_paperless_data
|
||||
paperless-ngx_paperless_postgres_data
|
||||
paperless-ngx_paperless_redis_data
|
||||
privatebin_privatebin_data
|
||||
ls: cannot access '/mnt/felhom-drives/scratch_hdd/felhom/backups/primary/': No such file or directory
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/loop1 69G 11G 54G 17% /var/lib/felhom
|
||||
/dev/nvme0n1 938G 210G 681G 24% /mnt/felhom-drives/scratch_hdd
|
||||
@@ -0,0 +1,104 @@
|
||||
10:10:26Z == 1 deploy
|
||||
10:10:27Z deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||
10:11:12Z state: running
|
||||
10:11:42Z == 2 marker A + backup
|
||||
10:11:46Z CREATE TABLE
|
||||
INSERT 0 1
|
||||
10:11:49Z [2] DB markers: 'A'
|
||||
10:11:49Z POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'}
|
||||
10:12:59Z backup finished: {"db_dump": {"count": 3, "duration": "1m9.055678483s", "last_run": "2026-10-20T10:12:58.690358712Z", "success": true}, "enabled": true, "running": false}
|
||||
10:13:04Z unit on disk:
|
||||
UNIT /mnt/sys_drive/felhom-data/backups/primary/rallly
|
||||
2026-10-20 10:12 487 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/app.yaml
|
||||
2026-10-20 10:12 4071 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/docker-compose.yml
|
||||
2026-10-20 10:12 11894 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/.felhom.yml
|
||||
2026-10-20 10:12 52228608 /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar
|
||||
2026-10-20 10:11 111371 /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql
|
||||
2026-10-20 10:12 3195 /mnt/sys_drive/felhom-data/backups/primary/rallly/manifest.json
|
||||
2026-10-20 10:12 1191 /mnt/sys_drive/felhom-data/backups/primary/rallly/data-stamps.json
|
||||
10:13:04Z == 3 POSITIVE CONTROL: marker B (live) then a normal unit restore
|
||||
10:13:09Z INSERT 0 1
|
||||
10:13:12Z [3-before] DB markers: 'A,B'
|
||||
10:13:12Z [3] restore points offered: [{"time": "2026-10-20T10:11:50Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}]
|
||||
10:13:12Z [3] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started']
|
||||
10:13:12Z [3] restore-status: {"running": true, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:13:12.817369857Z", "last": {"op": "offbox-reconstitute", "stack": "rallly", "ok": true, "message": "A(z) rallly: 0 fájl és 1 adatkötet és az adatbázis visszaállítva (mentés: 2026. 10. 20. 09:30) — az alkalmazás újraindult.", "finished_at": "2026-10-20T07:36:57.852528534Z"}}
|
||||
10:13:40Z [3] restore-status: {"running": false, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:13:12.817369857Z", "last": {"op": "restore", "stack": "rallly", "ok": true, "message": "A(z) rallly: 1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-20T10:13:39.161573115Z"}, "last_recent": true}
|
||||
10:14:00Z [3] after: state=running hold=None
|
||||
10:14:18Z [3-after] DB markers: 'A'
|
||||
10:14:18Z == 4 marker C (the newest live row), then THE FAULT: the unit's DB volume tar truncated to half
|
||||
10:14:22Z INSERT 0 1
|
||||
10:14:25Z [4-live] DB markers: 'A,C'
|
||||
10:14:29Z TRUNCATED /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar 52228608 -> 26114304
|
||||
10:14:29Z == 5 the unit restore over the damaged copy
|
||||
10:14:29Z [5] restore points offered: [{"time": "2026-10-20T10:11:50Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}]
|
||||
10:14:29Z [5] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started']
|
||||
10:14:29Z [5] restore-status: {"running": true, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:14:29.288204925Z", "last": {"op": "restore", "stack": "rallly", "ok": true, "message": "A(z) rallly: 1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-20T10:13:39.161573115Z"}, "last_recent": true}
|
||||
10:14:44Z [5] restore-status: {"running": false, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:14:29.288204925Z", "last": {"op": "restore", "stack": "rallly", "ok": false, "message": "Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1\nstderr: ...twork rallly_rallly-internal Creating \n Network rallly_rallly-internal Created \n Network rallly_rallly-internal Created \n Container rallly-postgres Creating \n Container rallly-postgres Created \n Container rallly Creating \n Container rallly Created \n Container rallly-postgres Starting \n Container rallly-p
|
||||
10:15:04Z [5] after: state=degraded hold=None
|
||||
10:15:38Z [5-after] DB markers: 'Error response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running\nError response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running\nError response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running'
|
||||
10:15:41Z containers:
|
||||
rallly Created
|
||||
rallly-postgres Restarting (1) 5 seconds ago
|
||||
10:15:44Z db container log tail:
|
||||
Data page checksums are enabled.
|
||||
|
||||
initdb: error: directory "/var/lib/postgresql/18/docker" exists but is not empty
|
||||
initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/18/docker" or run initdb with an argument other than "/var/lib/postgresql/18/docker".
|
||||
The files belonging to this database system will be owned by user "postgres".
|
||||
This user must also own the server process.
|
||||
|
||||
The database cluster will be initialized with locale "en_US.utf8".
|
||||
The default database encoding has accordingly been set to "UTF8".
|
||||
The default text search configuration will be set to "english".
|
||||
|
||||
Data page checksums are enabled.
|
||||
|
||||
initdb: error: directory "/var/lib/postgresql/18/docker" exists but is not empty
|
||||
initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/18/docker" or run initdb with an argument other than "/var/lib/postgresql/18/docker".
|
||||
10:15:47Z any undo / safety copy of the live state on disk?
|
||||
/mnt/sizing-bench/backups/secondary/paperless-ngx/recovery-unit/db-dumps/pre-restore-20260927T094411Z-paperless-ngx-postgres.sql
|
||||
/mnt/sys_drive/felhom-data/backups/primary/rallly
|
||||
/mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar
|
||||
/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql
|
||||
10:15:51Z controller log (restore lines):
|
||||
Container rallly Created
|
||||
Container rallly-postgres Starting
|
||||
Container rallly-postgres Started
|
||||
Container rallly-postgres Waiting
|
||||
Container rallly-postgres Error dependency rallly-postgres failed to start
|
||||
dependency failed to start: container rallly-postgres is unhealthy
|
||||
2026/10/20 10:14:42 manager.go:1362: [ERROR] [stacks] Stack rallly start failed after 1.1s: exit code 1
|
||||
stderr: ...twork rallly_rallly-internal Creating
|
||||
Network rallly_rallly-internal Created
|
||||
Network rallly_rallly-internal Created
|
||||
Container rallly-postgres Creating
|
||||
Container rallly-postgres Created
|
||||
Container rallly Creating
|
||||
Container rallly Created
|
||||
Container rallly-postgres Starting
|
||||
Container rallly-postgres Started
|
||||
Container rallly-postgres Waiting
|
||||
Container rallly-postgres Error dependency rallly-postgres failed to start
|
||||
dependency failed to start: container rallly-postgres is unhealthy
|
||||
2026/10/20 10:14:42 handlers.go:1881: [ERROR] [web] Restore failed (async): stack=rallly: starting rallly after restore from unit: starting stack rallly: exit code 1
|
||||
stderr: ...twork rallly_rallly-internal Creating
|
||||
Network rallly_rallly-internal Created
|
||||
Network rallly_rallly-internal Created
|
||||
Container rallly-postgres Creating
|
||||
Container rallly-postgres Created
|
||||
Container rallly Creating
|
||||
Container rallly Created
|
||||
Container rallly-postgres Starting
|
||||
Container rallly-postgres Started
|
||||
Container rallly-postgres Waiting
|
||||
Container rallly-postgres Error dependency rallly-postgres failed to start
|
||||
dependency failed to start: container rallly-postgres is unhealthy
|
||||
2026/10/20 10:14:49 manager.go:920: [DEBUG] [stacks] restart-policy of down member "rallly" = "unless-stopped"
|
||||
2026/10/20 10:14:59 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||
2026/10/20 10:15:09 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||
2026/10/20 10:15:09 main.go:4054: [INFO] [deadapp] rallly crossed the crash_loop threshold (8 in 10m0s) while a backup/quiesce/update holds it — NOT stopped
|
||||
2026/10/20 10:15:19 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||
2026/10/20 10:15:29 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||
2026/10/20 10:15:39 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||
2026/10/20 10:15:49 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||
exit 0
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
2026/10/20 10:14:29 handlers.go:1873: [WARN] [web] Restore requested (async): stack=rallly, snapshot=helyi from 172.18.0.3:36322
|
||||
2026/10/20 10:14:29 restore_dbcreds.go:114: [INFO] [backup] Restore rallly: database password(s) [DB_PASSWORD] come from the backup, [] stay as installed (older than the data) — values not logged (R-1106)
|
||||
2026/10/20 10:14:29 restore_unit.go:393: [INFO] [backup] Restoring rallly from recovery unit /mnt/sys_drive/felhom-data/backups/primary/rallly: images=2, secrets recovered=2/2, data_keys=0
|
||||
2026/10/20 10:14:29 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||
2026/10/20 10:14:40 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 10.8s)
|
||||
2026/10/20 10:14:40 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly
|
||||
2026/10/20 10:14:40 restore.go:246: [ERROR] [backup] Failed to populate volume rallly_rallly_postgres_data: tar: short read — exit status 1
|
||||
2026/10/20 10:14:40 restore_unit.go:448: [ERROR] [backup] volume restore for rallly: failed to restore 1 volume(s): [rallly_rallly_postgres_data]
|
||||
2026/10/20 10:14:40 [INFO] [stacks] SaveAppConfig: saved config for rallly
|
||||
2026/10/20 10:14:40 deploy.go:883: [INFO] [stacks] Redeploying rallly from recovery unit with 4 env vars
|
||||
2026/10/20 10:14:40 pin.go:93: [INFO] [stacks] pin rallly: rallly=lukevella/rallly:4.15.3, rallly-postgres=postgres:18-alpine
|
||||
2026/10/20 10:14:40 [INFO] [stacks] SaveAppConfig: saved config for rallly
|
||||
2026/10/20 10:14:40 restore_unit.go:467: [ERROR] [backup] rallly: NOT replaying the database copy — the volume restore failed, so the database is not known to be the copy's own
|
||||
2026/10/20 10:14:40 manager.go:1351: [INFO] [stacks] Starting stack: rallly
|
||||
2026/10/20 10:14:42 manager.go:1704: [ERROR] [stacks] Command failed: docker compose up -d (in /opt/docker/stacks/rallly) — exit code 1 (took 1.1s)
|
||||
2026/10/20 10:14:42 manager.go:1712: [ERROR] [stacks] stderr (last part): ...twork rallly_rallly-internal Creating
|
||||
2026/10/20 10:14:42 manager.go:1362: [ERROR] [stacks] Stack rallly start failed after 1.1s: exit code 1
|
||||
2026/10/20 10:14:42 handlers.go:1881: [ERROR] [web] Restore failed (async): stack=rallly: starting rallly after restore from unit: starting stack rallly: exit code 1
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
200
|
||||
{
|
||||
"ok": true,
|
||||
"message": "Stack rallly stop completed"
|
||||
}
|
||||
200
|
||||
{
|
||||
"ok": true,
|
||||
"data": {
|
||||
"removed": "rallly",
|
||||
"volumes_removed": [
|
||||
"rallly_rallly_postgres_data"
|
||||
],
|
||||
"hdd_paths_removed": [],
|
||||
"hdd_paths_preserved": [],
|
||||
"hdd_note": "Az alkalmazás nem tárolt saját adatot külső meghajtón, így ott nem volt mit törölni.",
|
||||
"userdata_kept": [],
|
||||
"backup_paths_removed": [
|
||||
"/mnt/sys_drive/felhom-data/backups/primary/rallly (26M)"
|
||||
],
|
||||
"verified": true
|
||||
},
|
||||
"message": "Stack rallly removed"
|
||||
}
|
||||
0
|
||||
0
|
||||
ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/rallly': No such file or directory
|
||||
/opt/docker/stacks/rallly
|
||||
@@ -0,0 +1,25 @@
|
||||
# C3 — R1/D4 proven live: a failed LOCAL restore leaves a broken app and no copy of what it replaced
|
||||
|
||||
Scratch guest 9202 on demo-hp, controller **0.331.0** (the local-restore code is unchanged since v0.328.0 —
|
||||
`git log` on `restore_unit.go`/`restore.go`; 0.336.0 runs the same lines). Hub off on 9202. The dashboard password was
|
||||
reset with the controller's own escape hatch (`--print-reset-code` → `/claim`), held in `sec` only.
|
||||
Architecture read: `07` §6.3 (the ladder „replay → rollback → hold"), §8; `09` §3 decision 154 (R-638 A), D20 (R-893 A).
|
||||
|
||||
| step (UTC) | act | observed |
|
||||
|---|---|---|
|
||||
| 10:10 | Rallly 4.15.3 + postgres:18 deployed through `POST /api/stacks/rallly/deploy` | running |
|
||||
| 10:11 | marker A (psql in the app's own DB container); „Mentés most" (`POST /api/backup/run`) | unit written 10:12 (`rallly_rallly_postgres_data.tar` 52 MB + `rallly-postgres.sql`) |
|
||||
| 10:13 | **positive control:** marker B, then the household's restore (`POST /backup/restore`, point „helyi") | „1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult"; markers **A** (B rewound, as designed) |
|
||||
| 10:14:22 | marker C — the household's newest row | live: **A,C** |
|
||||
| 10:14:29 | **fault:** the unit's DB volume tar truncated to half (a torn / damaged copy) | 52 228 608 → 26 114 304 B |
|
||||
| 10:14:29 | the same restore | stop 10.8 s → `volume rm` → `tar: short read` → `[ERROR] volume restore … failed` → definition rewritten → „NOT replaying the database copy" → start → **postgres crash-loops** (`initdb: directory … exists but is not empty`) |
|
||||
| 10:14:44 | what the household reads | „Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1 stderr: ...twork rallly_rallly-internal Creating …" — **raw English Docker output** |
|
||||
| 10:15:47 | any copy of the live state (marker C)? | **none** — no undo dump, no live-volume copy for Rallly (`find` of `*restore-undo*`/`pre-restore-*` finds only a September paperless file) |
|
||||
| 10:15 | the dead-app guard | `rallly crossed the crash_loop threshold … while a backup/quiesce/update holds it — NOT stopped` |
|
||||
|
||||
**So:** the household's live data (marker C) was destroyed by the restore itself, the app does not start, and nothing
|
||||
on the box can put the pre-restore state back. The off-site restore got exactly this protection in v0.336.0 (D20 = A:
|
||||
„put back exactly as it was"); the LOCAL path — 07 §3's main customer restore — did not. Logs: `01-run.log`,
|
||||
`02-controller-log-fault-window.txt`. Teardown through the product (`03-teardown.txt`): Rallly removed with its data
|
||||
and backups; no container, no volume, no unit (the `/opt/docker/stacks/rallly` folder is the catalog template every
|
||||
app has).
|
||||
@@ -0,0 +1,153 @@
|
||||
#!/usr/bin/env python3
|
||||
"""r1_proof.py — C3 proof of candidate R1/D4 on scratch guest 9202 (data-loss hunt 2026-10-20).
|
||||
|
||||
Claim under test: a LOCAL (Tier-1 unit) restore that fails after it has started replacing the app's volumes
|
||||
restarts the app on whatever is left, and keeps NO copy of the live state it replaced — so the household's newer
|
||||
data is gone and nothing can put it back (unlike the off-site path, which got „put back exactly" in v0.336.0).
|
||||
|
||||
Method (mechanism proof — markers are written with psql inside the app's own database container, as the
|
||||
2026-10-19 R-893 measurement did; the household-facing acts are the dashboard's own endpoints):
|
||||
1 deploy Rallly (throwaway) through POST /api/stacks/rallly/deploy
|
||||
2 marker A; „Mentés most" = POST /api/backup/run (whole box, scratch); wait
|
||||
3 POSITIVE CONTROL: marker B (live), unit restore (POST /backup/restore) -> A present, B gone (expected rewind)
|
||||
4 marker C (the household's newest live row); THE FAULT: truncate the unit's DB volume tar to half
|
||||
5 unit restore again -> observe: the answer, the app's state, what the DB holds, any undo/safety copy on disk
|
||||
6 teardown: remove Rallly with its data and backups through the product; verify nothing left
|
||||
Env: CTL_BASE, CTL_DOMAIN, CTL_PW_ENV (see tools/ctl.py). Prints a timestamped log (no secrets).
|
||||
"""
|
||||
import json, os, re, subprocess, sys, time
|
||||
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/rehearsal-2026-10-20/tools")
|
||||
import ctl # noqa: E402
|
||||
|
||||
APP = "rallly"
|
||||
DOMAIN = os.environ["CTL_DOMAIN"]
|
||||
|
||||
|
||||
def say(*a):
|
||||
print(time.strftime("%H:%M:%S", time.gmtime()) + "Z " + " ".join(str(x) for x in a), flush=True)
|
||||
|
||||
|
||||
def guest(script, timeout=600):
|
||||
r = subprocess.run(["ssh", "demo-hp", "export LC_ALL=C; cat > /tmp/r1p.sh; pct push 9202 /tmp/r1p.sh /tmp/r1p.sh;"
|
||||
" pct exec 9202 -- bash /tmp/r1p.sh; pct exec 9202 -- rm -f /tmp/r1p.sh; rm -f /tmp/r1p.sh"],
|
||||
input=script, capture_output=True, text=True, timeout=timeout)
|
||||
return (r.stdout + r.stderr).strip()
|
||||
|
||||
|
||||
def psql(sql):
|
||||
return guest(f"""C=$(docker ps --format '{{{{.Names}}}}' | grep -E '^rallly.*(postgres|db)' | head -1)
|
||||
U=$(docker exec $C printenv POSTGRES_USER); D=$(docker exec $C printenv POSTGRES_DB)
|
||||
docker exec -i $C psql -v ON_ERROR_STOP=1 -U "$U" -d "$D" -tA <<'SQL'
|
||||
{sql}
|
||||
SQL""")
|
||||
|
||||
|
||||
def stack():
|
||||
code, d = ctl.call("GET", f"/api/stacks/{APP}")
|
||||
return (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||
|
||||
|
||||
def wait_backup(t_cap=1200):
|
||||
t0 = time.time()
|
||||
time.sleep(5)
|
||||
while time.time() - t0 < t_cap:
|
||||
code, d = ctl.call("GET", "/api/backup/status")
|
||||
dd = (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||
if not dd.get("running"):
|
||||
return dd
|
||||
time.sleep(5)
|
||||
return {"timeout": True}
|
||||
|
||||
|
||||
def restore(tag):
|
||||
code, d = ctl.call("GET", f"/api/backup/snapshots?stack={APP}")
|
||||
data = d.get("data") if isinstance(d, dict) else None
|
||||
snaps = data if isinstance(data, list) else next((data[k] for k in ("snapshots", "items", "restore_points")
|
||||
if isinstance(data, dict) and isinstance(data.get(k), list)), [])
|
||||
say(f"[{tag}] restore points offered: {json.dumps(snaps, ensure_ascii=False)[:400]}")
|
||||
sid = snaps[0].get("id") or snaps[0].get("snapshot_id") or snaps[0].get("short_id")
|
||||
out = subprocess.run(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", ctl.HOST, "-H", "Cookie: " + ctl._SESS,
|
||||
"-X", "POST", "--data-urlencode", "_csrf=" + ctl._CSRF, "--data-urlencode", "stack_name=" + APP,
|
||||
"--data-urlencode", "snapshot_id=" + sid, ctl.BASE + "/backup/restore"],
|
||||
capture_output=True, text=True).stdout
|
||||
say(f"[{tag}] POST /backup/restore -> {out.splitlines()[0] if out else '?'} "
|
||||
f"{[l for l in out.splitlines() if l.lower().startswith('location')][:1]}")
|
||||
t0, last = time.time(), None
|
||||
while time.time() - t0 < 900:
|
||||
code, d = ctl.call("GET", "/api/backup/restore-status")
|
||||
dd = (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||
cur = json.dumps(dd, ensure_ascii=False)[:600]
|
||||
if cur != last:
|
||||
say(f"[{tag}] restore-status: {cur}")
|
||||
last = cur
|
||||
if not dd.get("running") and time.time() - t0 > 8:
|
||||
break
|
||||
time.sleep(3)
|
||||
time.sleep(20)
|
||||
st = stack()
|
||||
say(f"[{tag}] after: state={st.get('state')} hold={st.get('hold_reason')!r}")
|
||||
|
||||
|
||||
def markers(tag):
|
||||
say(f"[{tag}] DB markers: {psql('SELECT string_agg(m, \',\' ORDER BY m) FROM r1_marker;')!r}")
|
||||
|
||||
|
||||
def main():
|
||||
say("== 1 deploy")
|
||||
if not stack().get("deployed"):
|
||||
code, d = ctl.call("GET", f"/api/stacks/{APP}/deploy-fields")
|
||||
fields = (((d.get("data") or {}).get("metadata") or {}).get("deploy_fields")) or []
|
||||
vals = {"DOMAIN": DOMAIN, "SUBDOMAIN": "r1poll"}
|
||||
for f in fields:
|
||||
if f.get("env_var") in vals:
|
||||
continue
|
||||
if f.get("type") == "password":
|
||||
import secrets
|
||||
vals[f["env_var"]] = "R1-" + secrets.token_hex(10)
|
||||
elif f.get("required") and f.get("default"):
|
||||
vals[f["env_var"]] = f["default"]
|
||||
code, d = ctl.call("POST", f"/api/stacks/{APP}/deploy", {"values": vals})
|
||||
say("deploy ->", code, str(d)[:160])
|
||||
for _ in range(90):
|
||||
time.sleep(5)
|
||||
st = stack()
|
||||
if st.get("deployed") and (st.get("app_config") or {}).get("pinned_images") and st.get("state") in ("running", "unhealthy", "degraded"):
|
||||
break
|
||||
say("state:", stack().get("state"))
|
||||
time.sleep(30)
|
||||
say("== 2 marker A + backup")
|
||||
say(psql("CREATE TABLE IF NOT EXISTS r1_marker(m text primary key); INSERT INTO r1_marker VALUES ('A') ON CONFLICT DO NOTHING;"))
|
||||
markers("2")
|
||||
code, d = ctl.call("POST", "/api/backup/run")
|
||||
say("POST /api/backup/run ->", code, str(d)[:200])
|
||||
say("backup finished:", json.dumps(wait_backup(), ensure_ascii=False)[:500])
|
||||
unit = guest(f"for d in /mnt/felhom-drives/*/backups/primary/{APP} /mnt/sys_drive/felhom-data/backups/primary/{APP}; do "
|
||||
f"[ -d $d ] && echo UNIT $d && find $d -maxdepth 3 -type f -printf '%TY-%Tm-%Td %TH:%TM %s %p\\n'; done")
|
||||
say("unit on disk:\n" + unit)
|
||||
say("== 3 POSITIVE CONTROL: marker B (live) then a normal unit restore")
|
||||
say(psql("INSERT INTO r1_marker VALUES ('B');"))
|
||||
markers("3-before")
|
||||
restore("3")
|
||||
time.sleep(15)
|
||||
markers("3-after")
|
||||
say("== 4 marker C (the newest live row), then THE FAULT: the unit's DB volume tar truncated to half")
|
||||
say(psql("INSERT INTO r1_marker VALUES ('C');"))
|
||||
markers("4-live")
|
||||
say(guest(f"for d in /mnt/felhom-drives/*/backups/primary/{APP} /mnt/sys_drive/felhom-data/backups/primary/{APP}; do "
|
||||
f"[ -d $d ] || continue; f=$(find $d -name '*.tar' -path '*volume*' | grep -iE 'postgres|db' | head -1); "
|
||||
f"[ -n \"$f\" ] || f=$(find $d -name '*.tar' | head -1); s=$(stat -c %s \"$f\"); "
|
||||
f"truncate -s $((s/2)) \"$f\"; echo \"TRUNCATED $f $s -> $(stat -c %s \"$f\")\"; done"))
|
||||
say("== 5 the unit restore over the damaged copy")
|
||||
restore("5")
|
||||
time.sleep(30)
|
||||
markers("5-after")
|
||||
say("containers:\n" + guest("docker ps -a --format '{{.Names}} {{.Status}}' | grep -i rallly"))
|
||||
say("db container log tail:\n" + guest("C=$(docker ps -a --format '{{.Names}}' | grep -E '^rallly.*(postgres|db)' | head -1); docker logs --tail 15 $C 2>&1"))
|
||||
say("any undo / safety copy of the live state on disk?\n" + guest(
|
||||
"find / -xdev \\( -path /proc -o -path /sys \\) -prune -o \\( -iname '*restore-undo*' -o -iname 'pre-restore-*' \\) -print 2>/dev/null | head; "
|
||||
"for d in /mnt/felhom-drives/*/backups /mnt/sys_drive/felhom-data/backups; do find $d -maxdepth 4 -iname '*rallly*' -newer /etc/hostname 2>/dev/null | head; done"))
|
||||
say("controller log (restore lines):\n" + guest("docker logs --since 30m felhom-controller 2>&1 | grep -iE 'restore|rallly' | grep -vi 'GET /api' | tail -40"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user