100 lines
9.2 KiB
Markdown
100 lines
9.2 KiB
Markdown
# Part 3 — the data-loss candidates: research (night of 2026-10-22)
|
||
|
||
Research by a helper (read only; it changed nothing). Saved by the main session from the helper's report (the helper's
|
||
own write was refused by the harness). Baselines: controller cbc944c (v0.339.0), agent 2d5840b (v0.162.0), felhom.eu
|
||
1b442bcc. Architecture read: `07` §6.1.1, §10.2 (rows R-359, R-87), the design headers in R-899 `nightowed.go`,
|
||
`nightchain.go`, `offbox_clear.go`, R-822 `offbox_window.go`, hub F-14 `pbsdr.go`. Live reads: demo-hp's agent journal
|
||
(backup lines — `demohp-whole-box-completions.txt`) and `docker ps` on 9202. The main session's proofs and fixes are
|
||
recorded in the night report; this file is the input.
|
||
|
||
| Item | Verdict | Reason | Action |
|
||
|---|---|---|---|
|
||
| 1187a symlink | REAL (code) | `appexport/fabplan.go:292-310` follows a link; the tar write fails; `export.go:712` drops the whole drive folder; the export still ends „done" | fix + red-proof, P2 |
|
||
| 1187b owner/symlink | REAL (code) | `appexport/restore.go:1138-1183` handles only dirs and regular files, no chown | fold with (a) |
|
||
| 1187c stopped-app import | REAL (code) | `restore.go:355-374` removes the volumes only when the app RUNS; `docker cp` merges | 9202 proof, then refuse |
|
||
| 1187d | PLAUSIBLE | no `restoreOpBlocked` check in `StartImport` | fold into (c) |
|
||
| 1187f | REAL (code) | the bundle's HDD_PATH is used; an empty one gives a relative path | stays in the row |
|
||
| 1187g | REAL (code) | `classifyDBImage` does not know postgis (adventurelog, dawarich); psql without ON_ERROR_STOP | small fix, 9202 proof |
|
||
| 1188b | REAL in code, unlikely | `backup/restore.go:218` drops the rm error, then untars into the old volume | P4 |
|
||
| 1188c | ALREADY FIXED | v0.337.0, `TestR1188c_ALeftoverTempDoesNotBlockTheRetry` | close |
|
||
| 1188d | REAL (wording) | 30-min cap; scratch is kept, so a second press probably continues (not measured) | measure, then the sentence |
|
||
| 1188e | REAL, P4 | `offbox_restore.go:675-679` reads latest, `_ = id` | P4 |
|
||
| 1189a | REAL, half-live | agent `localapi/server.go:1179-1192`: a future archive time gives a negative age | agent fix needs the vouch |
|
||
| 1189b | SPLIT | the skipped nights after an off-cycle run: FIXED by R-899 (v0.304.0), shown in the journal; the ~3 min/night creep is still real | close that half; creep → P4 |
|
||
| 1189c | REAL in code | quiesce waits for the catch-up and the update leg, not for a restore | 9202 recipe |
|
||
| 1189d | REAL (code) | proof 05:30 and integrity 06:00 are daily jobs; the catch-up runs only the 3 backup legs; no hub age alarm | own row, ~15-line fix |
|
||
| 1189e | REAL (code) | the hub checks „since midnight"; any HH:MM window passes | hub row |
|
||
| 1189f | OPERATOR DECISION | design question | — |
|
||
| 1189g | REAL, P4 | spring-forward 02:30 → 03:30 collides with tier2 | unit test |
|
||
| 1189h | not re-read | — | stays |
|
||
| 1189i | REAL, P4 UX | check and begin are not atomic (`web/handlers.go:1860/1881`) | P4 |
|
||
| 1190a | REAL (code) | the ceremony stages the off-site password only if off-site is configured; hub „abc"→"" demotes the good package | unit tests, ~6-line fix |
|
||
| 1190b | REAL (code) | the global bar ignores `recoveryOffer()`; the start handler does not refuse | 409 fix, P2 |
|
||
| 1190c | PLAUSIBLE | — | read-only fingerprint compare for Tester4 |
|
||
| 1190d | OPERATOR DECISION | — | — |
|
||
| 1190e/f/h | not re-read | — | stay |
|
||
| 1190g | REAL (code) | a nil AppConfig drops the app from `liveDriveBinds`, so its live folder can be listed (and deleted) as kept | fix, P2 |
|
||
| 1205 | OPERATOR DECISION | the F-14 gate is deliberate (ruling 2026-07-13) | two options to the operator |
|
||
| 1206 | no loss on the Felhom tier (by design, R-822); the NAS tier open | the guard refuses a prune while any snapshot is future-dated; the NAS tier prunes without the guard | pure unit test with the policy simulator |
|
||
|
||
## R-1187(c) — a `.fab` import into a STOPPED app merges into the old volumes. REAL (code)
|
||
|
||
`controller/internal/appexport/restore.go:355-374` removes the volumes (`RemoveStackVolumes` = `compose down --volumes`,
|
||
`cmd/controller/main.go:3566-3591`) only when `IsStackRunning` (`main.go:3467-3471`: Running or Degraded). Every other
|
||
state (household-stopped, crashed, error, starting) „overwrites config in place". `restoreVolumeData` (`:736-772`) then
|
||
runs `volume create` (a no-op on an existing volume) and `docker cp -` (`:717-732`), which adds and overwrites but
|
||
never deletes: the result is old ∪ bundle. The drive half (`restoreHDDData` `:590-648`, `extractTar`) merges in both
|
||
branches. Harm: a broken app (not running) is the likeliest import target; a stale SQLite `-wal` beside the bundle's
|
||
`.db` is a corruption shape. „No SQL dump by default" alone is NOT-A-FINDING: with stop ticked the DB volume is captured
|
||
cold, which is consistent (`export.go:258-279`, `:366`; `appbackup/dbdump.go:142-144` lists only running containers).
|
||
Fix: `executeImport`, ~8 lines — (B) refuse an import over an installed app that is not running (now); (A) remove its
|
||
volumes too, only together with the D22 put-back copy. 9202 recipe (~20 min): a throwaway uptime-kuma; MARK-A into the
|
||
volume, export (stop ticked), MARK-N; control: import while RUNNING → MARK-N gone; MARK-N again, Stop, import → MARK-N
|
||
survives + the debug line „deployed but not running — will overwrite config in place".
|
||
|
||
## R-1189(b) — split
|
||
|
||
The skipped nights after an off-cycle run are ALREADY FIXED by R-899, controller v0.304.0 (2026-10-09),
|
||
`quiesce/nightowed.go`, `TestR899_*` (6). Proven on demo-hp: before the fix every off-cycle run was followed by a
|
||
skipped night (10-01 22:10 → no 10-02; 10-05 11:19 → no 10-06; 10-08 22:20 → no 10-09); after it every one was followed
|
||
by a night run (10-17 16:33 → 10-18 04:33; 10-19 03:25 → 04:32; 10-20 08:42 → 10-21 04:32). The creep is still REAL,
|
||
low harm: agent `localapi/server.go:1184` is `age >= Cadence`, age counted from the archive START; the controller polls
|
||
every 5 min (`quiesce.go:150`) and the archive starts after the apps are stopped. Measured with no press between:
|
||
04:31:07 → 04:33:18 → 04:36:47 → 04:38:14, and 04:33:37 → 04:38:44 → 04:42:40 (~1.5–5 min/night). Unpressed, the start
|
||
reaches W+6h in ~70–80 nights, then one night is skipped (44 h is below the 48 h valve, `quiesce.go:807-818`); the hub's
|
||
26 h line (`deadline.go:29`) alarms the operator. Fix: `quiesce/tiers.go` — inside the window a tier is due when
|
||
`age >= cadence - 4h` (~6 lines, no customer data); proof: a unit test with an injected clock.
|
||
|
||
## R-1189(d) — proof and integrity never run on a box that is off at 05:30–06:00. REAL (code)
|
||
|
||
`main.go:1613` `Daily("offsite-integrity","06:00")`, `:1635` `Daily("offsite-proof","05:30")`; the scheduler picks the
|
||
next future time (`scheduler.go:499-518`) and skips late fires (`:341-346`); the catch-up runs only the three backup
|
||
legs (`main.go:1544-1553`, `catchUpLegs` `:4138`). The comment at `main.go:1601-1603` („checked the next day it is on")
|
||
is true only if the box is on at 06:00. The hub shows `offsite.last_integrity_*` (`web/offsite_box.go:199-210`) with
|
||
no age alarm. Design: `07` §10.2, R-359 („due-ness, not a weekday"). Fix: an After-hook on `nightchain.CatchUp` running
|
||
the due-gated integrity check and proof once; the comment corrected (~15 lines); optional hub age alarm. Red-proof: a
|
||
CatchUp test with a never-05:30 clock and a recording hook.
|
||
|
||
## R-1190(a) — a ceremony with off-site not configured seals no repo password and supersedes the good package. REAL (code)
|
||
|
||
`web/escrow_handlers.go:217-224` stages only if `OffboxConfigured()` (`backup/offbox.go:695-710`) — false when the target
|
||
is cleared or disabled, although `ClearOffboxTarget` deliberately keeps the password file (`offbox_clear.go` header).
|
||
The agent then seals no password (`cmd/felhom-agent/main.go:2981-3003`) and uploads sha "". Hub `SaveHostEscrow` sees
|
||
`curSHA != ""` and demotes the good package to retained custody (operator-only). `OffsiteRecoveryOffer` reads "" as
|
||
„covers no key" (`offbox.go:1855-1873`) and says nothing. Fix: stage when the password file exists (~6 lines;
|
||
`PushOffboxPasswordForEscrow` `offbox.go:800` reads only the file). Red-proof: a web test asserting the stage fn is
|
||
called; a hub store test pinning „abc"→"" ⇒ superseded.
|
||
|
||
## Other mechanism lines
|
||
|
||
1187a `fabplan.go:292-310` (a FIFO also hangs the export on `os.Open`) · 1187b `restore.go:1138-1183` · 1187g
|
||
`restore.go:1047-1056`, `933-943` vs `appbackup/dbservices.go:37` (the dump is `--clean --if-exists`, so ON_ERROR_STOP is
|
||
safe; copy `dbdump.go:775-784`) · 1188b `backup/restore.go:218` (`StopStack` is `compose down`, `stacks/manager.go:1444`)
|
||
· 1188d `web/offbox_handlers.go:455` (scratch kept, `offbox_restore.go:373-392`) · 1188e `offbox_restore.go:675-679`,
|
||
marker SnapshotID `:458` · 1189c `quiesce.go:213-275`, `:565` · 1189e hub `monitor/deadline.go:416-419`, `555-565` ·
|
||
1189i `web/handlers.go:1860`, `1881` · 1190b `web/escrow_banner.go:23-35`, `templates/backups_remote.html:144-160`
|
||
(its comment names the harm, R-199), `escrow_handlers.go:180-245`, reachable in offer shape (c) `offbox.go:1866-1870` ·
|
||
1190g `stacks/kept.go:220-235`, `336-360`, `372-389`; `deploy.go:1113-1125` (a parse error gives nil) · 1205 hub
|
||
`web/pbsdr.go:236-248`, `store.go:3293-3297` (deliberate; operator ruling 2026-07-13) · 1206 `backup/offbox_window.go:22-24`,
|
||
`130-137`.
|