155 Commits

Author	SHA1	Message	Date
admin	d698ce343b	controller v0.242.0: a removed app is listed with its kept backup; five small ones (R-487 R-491 R-490 R-489 R-476 R-456) ... R-487: the local backup lists are keyed on the drives, not on what is deployed — a removed app whose unit was kept is listed with the restore that reinstalls it, the picker answers for it, and the restore opens the unit where it sits. R-491: a removal clears the app's update hold. R-490: /api/system/info reaches the API router and reads the default storage path. R-489: volumes_removed is the real before/after difference, [] when none. R-476: a Tier-2 copy is dated by its data, not its manifest. R-456: the boot-orphan rule is pinned. Every fix red-proofed.	2026-09-13 22:50:18 +02:00
admin	3e813307cc	controller v0.241.0: a bind-data app leans on off-site before its own unit; the hold names what the copy holds (R-479) ... Operator ruling 2026-09-13. An app with classified binds walks second drive -> off-site -> own unit (its unit holds no files); volume apps keep 2 -> 1 -> 3. RestoreHold.CopyHolds records what the chosen copy holds and the sentence ends with it; older holds keep their tier-only sentence. Tests on both halves; red-proof: a layout-blind order fails the bind case.	2026-09-13 21:47:33 +02:00
admin	b93c1543da	controller v0.239.0: any backup tier lets an app update (R-475) ... Operator ruling 2026-09-13. The update precondition walks Tier 2, Tier 1 (own recovery unit, "helyi") and Tier 3 (off-site, 15 s bound; unreachable counts as absent with a WARN) and leans on the first FRESH copy; the backup_max_age rule applies to whichever tier is chosen. No copy anywhere: back up first. Refused only when nothing exists and no backup can be taken. RunAppBackupNow tolerates a Tier-2 failure (WARN) and marks the captured unit proven current. The hold names the tier (második meghajtó / saját meghajtó / távoli mentés) and the date; pre-v0.239.0 holds keep their text. A successful off-site restore now lifts an update hold. The backups page still uses Tier2UnitRestorePoint unchanged. Scenarios G-M tested; red-proofs M, L, the tail and the off-site clear in felhom.eu documentation/audits/rulings-r472-r475-2026-09-13/.	2026-09-13 17:16:24 +02:00
admin	cbcca03061	v0.238.1: the nightly backup leaves an app alone WHILE it is being updated, not only once it is held (slice 4 follow-up) ... Found live in v0.238.0 Scenario F on demo-hp: during an update's 5-minute health wait the app is not yet held, and the periodic recovery-unit capture at 10:17:09 wrote the never-started definition (alpine:3.20) into its PRIMARY unit, 53 s before the hold landed. The Tier-2 mirror the hold names survived only because Tier 2 runs daily; a nightly Tier 2 inside a verify window would have mirrored the broken definition over the copy the customer is told to restore from. backup.Manager.isHeld — consulted by the capture sweep, the Tier-2 run and the volume dump — is now also true while a guarded update is moving the app, via SetUpdatingCheck wired in main.go to stacks.Manager.IsUpdating. Test with positive control + red-proof; wiring pinned. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS	2026-09-13 12:25:09 +02:00
admin	0d402f711d	v0.237.0: the Update button takes a backup first, and tells the truth (update arc slice 4 — R-448, R-443, R-439) ... POST /api/stacks/{name}/update is now a guarded job answering 202: cheap refusals (hold — R-439, busy, migration, deploying, memory via the deploy's own memoryVerdict, a fixed 2 GB disk floor, and no restorable Tier-2 copy) → backup-first when the proven copy is older than update.backup_max_age (24h) → safety dump BEFORE the pin moves → pin → pull (failure puts the pin back) → up → health (.felhom.yml check or 60 s settle, update.health_timeout 5m). Not healthy → the app is stopped and HELD (RestoreHold reason update_failed, same store and gate as R-379) and the page names the backup to restore from; the pin stays. Success is only ever update_phase=done after health (R-443). UpdateStack is deleted. The restorable-unit predicate is EXTRACTED to backup.Tier2UnitRestorePoint and shared with the backups page (row pinned unchanged). The copy is aged by the last successful Tier-2 copy, not the manifest created_at — measured on demo-hp that created_at moves only on definition changes. Crash safety: update-journal.json before each phase; RecoverUpdates before the boot sweep, ResumeInterruptedUpdates after the guards are wired. Three unattended start paths ignored a hold and now honour it: the drive-return gate (restart + boot recreate) and the nightly volume dump. The nightly capture and Tier-2 run skip held apps so the restore point survives. No automatic rollback — measured per-app; route back = restore. Tests A–H across stacks/backup/api/web/cmd; six red-proofs seen to fail. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS	2026-09-13 11:41:31 +02:00
admin	8a0e0a59ad	v0.235.0: freeze the version, keep the fixes flowing (operator ruling 2026-09-06) ... Slice 3. R-447 was BLOCKED because R-438 established that RestartStack's use of up -d to pick up template changes was CHOSEN and written down in its own comment. The operator ruled Option 1, and this implements it. The rule: while the catalog offers the same version you run, its fixes flow to you; the moment it moves to a newer version you are frozen until you update. NOTHING was added to any of the thirteen compose up -d call sites. Most of them are repairs - the boot reconciler, the drive-return gate, the app-stop guard - and a repair path that refuses to repair leaves a customer's app down, which is worse than the problem. They are made safe by removing the reason. app.yaml gains pinned_images: what the app is SUPPOSED to run. It is NOT installed_images, which is an observation; letting a reading become a deployment is the R-166 category error one field over. Four writers, each also storing the exact definition as applied-compose.yml. UpdateStack advances the pin and re-renders BEFORE the pull, because pull and up -d act on the file on disk, and a pin set afterwards would pull the frozen version and report success. The syncer renders instead of copying, through one nil-safe seam. Catalog images equal the pin -> verbatim, so fixes and self-healing both survive; they differ -> the WHOLE stored definition, never a substitution of refs into a newer template (wger 2.6 needs a DB config the older template cannot supply). This is deliberately not 'skip deployed apps', which was option B and was rejected. AdoptPins runs once at boot after the backfill, files only, and skips loudly rather than inventing a pin. syncer.Start() moved to after it: the initial sync would otherwise run while every app was unpinned and overwrite a deployed app's version once per boot. THE BADGE HAD TO CHANGE OR SLICE 2 WOULD HAVE INVERTED SILENTLY. TemplateImages reads the LIVE compose file, which is now the frozen one, so the comparison would have answered Naprakesz on exactly the apps that are behind - with every test green, because the new field has the same type. It now reads CatalogImages. +16 tests (1729 -> 1745), 28 packages green. Three red-proofs run and reverted. A test also caught the syncer writing an empty compose file over a live app.	2026-09-06 09:45:34 +02:00
admin	38d28b5b62	v0.234.0: seed installed_images at startup, so the label appears on an app nobody touched ... The operator looked at demo-felhom the morning after v0.233.0 and found OpenGist - up 15 hours, running exactly the catalog pin - showing no badge at all. v0.233.0 wrote the record only from the four bring-up paths, so an app nobody restarts carried no record indefinitely. On a quiet box that is every app, which is the box we most want to see. The known limitation WAS the feature not working. BackfillInstalledImages runs once at startup, beside BackfillDesiredState and before the boot reconciler. It READS containers: starts nothing, restarts nothing, writes no compose file. It never overwrites an existing record. And it REFUSES to seed a partial observation, which is why this is not a three-line loop: the badge reads a service-count mismatch as BEHIND, so seeding a degraded app from what is visible would render 'Frissites elerheto' over an app that is perfectly current. The bring-up paths may write a partial because they follow a successful up -d where a gap is real news; a backfill meets any state. Same data, two writers, two admission rules - deliberately. Also fixes a calendar bomb of mine: the render test hardcoded catalog_since and the string '46 napja', but the render path reads time.Now(), so it was green on the day it was written and red the next morning. Now derived. Filed as R-457 with six other candidate files named as unchecked, not accused. +5 tests (1724 -> 1729), 28 packages green. Red-proof of the partial guard run and reverted; the wiring and its ORDER pinned by an AST walk.	2026-09-03 11:56:43 +02:00
admin	303129e3af	v0.231.0: the off-site proof gets a by-hand trigger, like its integrity sibling (R-87) ... Without it the only way to see the job work is to wait for 05:30, which makes live validation and any future diagnosis a next-day exercise. Same function as the scheduled job - no second code path. ONE deliberate difference from the integrity button: due-ness is NOT bypassed. There, forcing means "check the store again", which is always answerable. Here due-ness IS the target selection - an app is due when its newest snapshot has not been proved - so ignoring it would mean inventing a second way to choose an app, exactly what having one function prevents. When nothing is due the button says so, honestly. Every other guard intact, including the single-writer flag: a hand-run during a backup SKIPS exactly as the scheduled one would. POST /api/debug/backup/offsite-proof, button beside "Restic integritas" on the debug page. debug_route_gate pairs the two, so a button with no dispatch (R-400's shape) cannot ship.	2026-08-31 21:09:09 +02:00
admin	e43b5ec07d	v0.231.0 - the box proves its own off-site copy still holds something (R-87) ... R-87 re-scoped by its own spike and built as Option C. MinAgent 0.129.0 unchanged. THE QUESTION NOTHING ASKED. The weekly check proves the stored bytes are the bytes we stored; it cannot tell us we stored the WRONG thing. A hollow recovery unit backs up cleanly, checks cleanly at 100 percent depth, restores cleanly and gives the customer nothing back - measured on demo-hp 2026-08-31, 120082104 B to 7036 B in one nightly run recorded as a success (R-403). No tier and no cadence asked it. Now offsite-proof does, nightly, on one app. IT DOES NOT prove a restore puts data back into a running app. That stays drill work and 07 section 8 matrix row 4 is NOT moved. THE ACCEPTANCE RULE HAS TWO PARTS AND THE OBVIOUS ONE IS A TRAP. "Check the unit against its own packing list" PASSES a hollow unit, because a hollow unit declares nothing. So: (1) everything declared is present, AND (2) the manifest declares what the app is supposed to have. Part 2 is the whole value. RED-PROOFED: the naive rule makes the hollow-unit test read verdict "pass". THE EXPECTATION COMES FROM INSIDE THE UNIT, never the live box - the snapshot may predate the app's shape, and GetDockerVolumes describes the running app. Database half is DBServiceNames, the same discriminator RestoreFromRecoveryUnit uses. Volume half is ParseComposeNamedVolumes as an EXISTENCE check, not a name match: tars are <project>_<volume>.tar and ResolveDockerVolumeNames derives the project from the compose file's parent dir, which inside a unit is the literal string "compose". Measured on all eight real units on demo-hp the counts match exactly and the naming held every time - but "held on eight" is not "derivable" (R-355). Half a rule that is true beats a whole rule that is invented. THREE OUTCOMES: pass, fail (readable and empty), cannot judge. An app that legitimately has neither a database nor volumes PASSES. RED-PROOFED: alarming on any empty unit makes that test read verdict "fail". IT NEVER WRITES TO THE REPOSITORY and that is asserted on the ARGV as a non-effect: --no-lock, no unlockStale, and m.runner() rather than resticStep so the unlock --remove-all escalation is unreachable. RED-PROOFED: routing it the customer path's way makes the test fail on "unlock" appearing in the argv. IT TAKES acquireRunning ITSELF and skips rather than waits, because RestoreOffboxScratch does not take it (R-408) while offbox_integrity.go states that invariant as universal. DUE-NESS IS PER SNAPSHOT (R-86's model), never per clock. RED-PROOFED: recording a timestamp fails the stored-value test AND breaks the rotation - night 2 re-picks night 1's app. ITS SCRATCH IS A SEPARATE ROOT (backups/offsite-proof) and that is a safety decision, not tidiness: the job deletes its copy on every path, and sharing backups/offsite-restore/<app> would mean a nightly background job deleting the verification copy a CUSTOMER is looking at. It is also invisible to placement, so a proof copy can never be pushed into a live app. SHARED RATHER THAN FORKED: offboxScratchDirIn parameterises the scratch resolver on its ROOT builder, and unitOnlyHeadroom extracts the free-space gate, so the customer path and the proof refuse at the same floor with the same Hungarian sentence. RestoreOffboxScratch's behaviour is unchanged. NEW EVENT offsite_proof_empty, severity error, operator-only - deliberately NOT backup_integrity_failed, whose hub template says the store is DAMAGED. Here the store is sound and the content is absent: different cause, different action. The hub half shipped FIRST, in felhom.eu 1aeaa30 (hub v0.110.0, live and verified), because an unallowlisted type is 400'd and vanishes. 33 new tests, all groups green; full suite 1689 tests, 28 packages, rc=0. All 13 controller gates OK. Five red-proofs run and recorded in REPORT.md. A golden carrying 0.231.0 is OWED - the fleet is on 0.230.0. Viktor's call (R-242).	2026-08-31 20:55:34 +02:00
admin	3c49dc8ea4	v0.228.0 — the off-site check reads the data; the debug page stops lying (R-399 + R-400) ... R-399: monitoring.integrity.read_data_subset defaults to 100%. A pack damaged without changing its size made plain `restic check` report "no errors were found" on demo-hp 2026-08-30; every read-data form caught it. Cost on that 134 MB store: 35.0s structure vs 39.2s at 100%. "off" (any case) is the off token; empty means not-configured, therefore the default; a malformed value falls back to the DEFAULT, never to structure. A completed check over 5 minutes logs a WARN naming the duration, the depth and R-401 — operator log only, no hub event, no depth change. The depth is now recorded with the verdict (LastIntegrityDepth; empty = NOT RECORDED, never "structure"). R-400: 24 debug-page references, 17 dispatched, 7 dead — three of which fetched on page LOAD, so those panels were permanently blank. backup/crossdrive implemented; backup/infra, hub/infra-push, dr/infra-status, storage/watchdog-status and both storage/simulate-* deleted with their panels and JavaScript. scripts/debug_route_gate.py fails in both directions and is registered after the seven were resolved. 18 referenced, 18 dispatched, none orphaned. Corrections: the dead-field warning in report/types.go said the controller runs no integrity check and the notifiers are called from nowhere — both false since v0.227.0. controller.yaml.example gains its missing integrity: block. integrityCheckTimeout's "ships OFF" comment rewritten.	2026-08-31 10:24:29 +02:00
admin	0d52a42c17	R-359 + R-397: the off-site store gets checked, and the advertised check becomes real ... Nothing ever verified that the off-site copies are still readable. The whole-guest tier has verify jobs; the tier holding the customer's documents and photos had none -- the complete set of restic verbs this controller used contained no `check`. We would have found out at restore time, with a customer waiting. On 2026-08-21 a deliberately damaged pack was caught at once by plain `restic check`; we had never run it. R-397: NotifyIntegrityOK/NotifyIntegrityFailed existed with no caller, the hub allowlists both event types and carries the Hungarian text for both, the settings checkbox exists, and the debug button posts to /api/debug/backup/ integrity. Everything was built except the part that runs. SIXTH instance of that shape in this project. THE HAZARD SHAPES THE WHOLE DESIGN. resticStep self-heals a crash lock by running `unlock --remove-all` and retrying, and its own comment records why that is safe: every caller holds the in-process single-flight mutex, so any lock it meets is stale. A check that did not take that flag could meet a LIVE prune's lock from this same box, remove it, and retry over the top of it. So the check TAKES THE FLAG and SKIPS rather than waits -- waiting would pin the nightly backup behind it, and a skip costs nothing because due-ness makes tomorrow try again. TestR359_SkipsWhenRunningFlagHeld asserts the NON-EFFECTS: restic never invoked, `unlock` never in any argv. Its red-proof prints the real thing -- restic running `check` while the flag was held. DUE-NESS, NOT A WEEKDAY. Daily job, weekly behaviour: "is the last successful check older than 7 days?" not "is it Sunday?". R-341 is exactly the other shape, a dated check quietly missed and never caught up. No Weekly primitive added. THREE OUTCOMES, NOT TWO. Skipped, Unreachable and failed are different facts. "I could not look" is not "I looked and it is broken" -- R-339 already owns reachability, and a second alarm for the same fact trains the operator to discount the one alarm that means the backups are damaged. A timeout is unreachable, never damage. A failure advances due-ness (a broken store must not be re-checked nightly); a skip and an unreachable store do not. Success is severity `info`, which severityNotifies DROPS -- it mails NOBODY, by design. A weekly success e-mail is how people stop reading their alerts. The customer gets a SENTENCE; restic's words go to the log, truncated (R-379: 615 bytes of raw database text reached a customer once). read-data-subset ships OFF and a malformed value is refused at read time rather than handed to restic, where one typo would fail the whole check. Published on OffboxReportStatus, NOT on report.BackupReport's IntegrityOK -- those were retired by R-331 YESTERDAY and TestBackupReport_DeadFieldsStayZero still passes unmodified. Also: the monitoring page stopped promising a Sunday job that never existed, and the debug button got its dispatch case. PART 0 WAS NOT BUILT, AND R-398 WAS MY OWN MISTAKE. The seam it asked for already exists: offboxRunner/SetOffboxRunner/m.runner() has been injectable since the off-site tier shipped, and other tests drive restic-backed paths through it. A resticStepFn seam would have been WORSE here -- it would replace the `unlock --remove-all` escalation and hide it from the assertions that must see it. R-358's AST ordering test is converted to a real execution test instead, which immediately surfaced something the AST walk could not: unlockStale legitimately runs before the restore. Four red-proofs, each printing the pre-fix behaviour. Green gate: 28 packages, rc 0. All 12 controller gates OK.	2026-08-30 21:03:29 +02:00
admin	92cebb8c95	R-330: stop the backup alarming about the apps it is holding down (v0.224.0) ... Measured live on demo-hp 2026-08-30 (controller 0.223.0): the nightly db-dump and offbox-backup legs stop each stack ~13s to tar its volumes while the deadapp-check job scans every 30s, so the scan caught whichever stack was mid-cycle and pushed app_start_failed to the customer. 61 e-mails about apps that were never broken. The defect is not a missing mechanism. quiesce/suppress.go solved exactly this in v0.179.0 and works -- but classifyRunStates read only the quiesce loop's set, and that loop covers the WHOLE-GUEST backup. The per-app legs stop stacks through Manager.DumpAppVolumesSafe, which registered with nothing. Two mechanisms stop apps on purpose; only one told the alarm. Fifth instance of the "seam built but never wired" class, and the first where the unwired half was a consumer. The suppression now rides AppStopGuard, which already brackets every deliberate stop in the product (Begin before the stop, End after a successful restart) at all three call sites, and which main.go hands as ONE object to the backup manager and the exporter. scanDeployedAppRunStates takes the union of both sets. All three per-app stop paths are covered, not only the reported nightly one. It cannot latch -- End() runs only on a restart that SUCCEEDED, so unlike the quiesce loop an open-ended hold is a real hazard here: 1. ReleaseFailed drops the entry IMMEDIATELY on a restart that broke, wired at every failure path, so the app alarms on the next scan; 2. Begin REPLACES the set (one marker file = one operation); 3. appStopMaxHold (6h) caps a hold nothing released, logged at WARN. Grace is 180s, deliberately quiesce's own constant and derivation. Suppression is NOT persisted: after a crash the guard holds nothing and a down app must alarm. ReleaseFailed keeps the durable crash marker; a test pins that. Three companion red-proofs, each printing the pre-fix value (REPORT.md section 5): - drop markStopped from Begin -> "suppressed at stop = map[]" - drop ReleaseFailed from the dump -> "map[bookstack:true] after a restart that FAILED" - pass nil instead of appStopGuard -> the AST wiring test fails The third is load-bearing: the component was never the broken part, so a suite that only injected it would have been green against the shipped defect. Green gate clean: go build + go vet + go test ./... -- 28 packages, rc 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LB8FmJaGd2cyjvy6dbEjpM	2026-08-30 17:58:15 +02:00
admin	9832760027	v0.223.0: the app-down alarm reached nobody (R-329), and the stop nobody heard (R-386) ... R-329. NotifyAppStartFailures emitted severity "warn". The hub accepts exactly {info, warning, error, critical} and silently coerces anything else to "info", which severityNotifies then drops BEFORE both legs. Banner shown, event stored, POST 200, no mail sent. One word. This is the second time: DiskAlertKind.Severity emitted "warn" until v0.215.0 and its own comment records that every warning-level disk alert went to nobody. A comment recorded the lesson and nothing enforced it. The guard is now an AST walk over the whole controller - grep cannot work here, since "warn" appears legitimately nine times as a healthcheck status vocabulary. The sweep found exactly one bad severity. Its limits are stated: the walk cannot follow a variable, so all six dynamic call sites are registered by name with the values each can take, and a new one fails the test. Two of the six were found by the guard, not by the hand sweep before it. Also pinned: fillwatch.Band.Severity() returns "" for BandOK, which would vanish the same way. It is unreachable because Check() notifies only on escalation - but that safety lives in a different function from the one that looks unsafe, so the test asserts the consequence rather than the mapping. app_start_failed gains a customer toggle, DEFAULT OFF, per operator ruling. The operator is mailed either way: processOperator never consults customer prefs. It is deliberately NOT in operatorOnlyEvents, which would make the toggle a lie. R-386. classifyRunStates decided "the customer stopped this" from the STATE, so every stopped stack was assumed deliberate. Measured on demo-hp: privatebin stopped out of band, nine scans, zero events, zero banner - while the comment beside it claimed an out-of-band stop still alerts. DesiredState already records the answer and has exactly one writer. Stopped -> no alarm; Running -> alarm; absent -> UNKNOWN, keep today's behaviour AND say so. Absent stays silent deliberately: reading it as "nobody asked" would email about every app anyone ever stopped, fleet-wide, on the first cycle after upgrade. The gap is bounded not silent - IntentUnknown is set and the names are logged at INFO on the heartbeat cadence. failedRestart still lifts a Stopped intent, or F-CRIT-1 re-opens. No new DesiredState writer. Two settings toggles each governed two alarms. "Lemez figyelmeztetes (90%+)" also wrote disk_critical, the drive-is-FAILING alarm. Now four honest toggles; 12 became 15. A no-op save stores the existing slice verbatim, so byte identity is by construction - without that guard the defaults case reorders, which the red-proof caught. Test count 1504 -> 1522. Five red-proofs, five seen failing; one passed first time and is reported - that mutation was inert, not the test weak.	2026-08-23 11:21:06 +02:00
admin	5da11c4480	v0.222.0: ask whether a supervised member is DEAD before whether one is UNHEALTHY (R-384), and stop promising an undo copy nobody looked for (R-383) ... R-384. aggregateState returned StateUnhealthy the moment unhealthy > 0, and the R-51 mixed-case block that asks "is a supervised member dead?" sat below it. A two-container app whose database exits goes unhealthy BECAUSE it cannot reach that database - so the symptom the dead database causes was what suppressed the alarm for it. unhealthy is not a down state, so classifyRunStates never marked the app down and app_start_failed never fired. Measured live on demo-hp 2026-08-22: bookstack-db stopped at 21:27:01 and the F-OBS heartbeat printed "0 currently down" throughout. R-51's 18-hour immich failure, back through a different door. Two things moved, and either alone leaves the defect standing: the supervised test is hoisted above the unhealthy/starting/restarting returns, and "some members are up" now counts ANY member not in the down bucket. The old guard was running > 0, which made the R-51 block unreachable in exactly the case it was written for. IsDownState is byte-identical - unhealthy stays excluded, because an unhealthy container is running and folding it in reintroduces the flapping that exclusion exists to stop. No new state was minted. Only the ORDER changed. The priority comment was rewritten because it asserted an ordering the code no longer has. Three subtests in TestAggregateState_UnchangedBranches were AMENDED: they asserted an unhealthy/starting/restarting member beat an exited peer on unless-stopped, which pinned the defect as settled behaviour. They keep their intent with the down member given a benign policy. R-383. The double-failure message said the previous state's backup EXISTS, built from the returned path without asking the filesystem - and a missing file is one of the two ways that rollback fails. undoCopyPhrase now describes the copy from disk: present, partial, missing (still naming where it should be), or never written. Zero-length counts as missing. Test count 1494 -> 1504. Four red-proofs planted, four seen failing; the two halves of R-384 convict independently.	2026-08-23 07:25:00 +02:00
admin	968c968559	R-361: the safety dump destroyed the app's own database backup ... writeSafetyDump called DumpOne into the app's OWN unit dir and renamed the result to pre-restore-* afterwards. DumpOne writes <stack>-<dbtype>.sql - the app's canonical dump - so every safety dump overwrote the app's real backup and then moved it away, leaving the app with no database backup until the next nightly run. A local restore-from-unit in that window tells the customer the app never had a database. The comment beside it asserted the rename meant it 'can never overwrite the app's real dump'. False as written, and believed for four months. Measured live before the fix: docmost and bookstack each held only pre-restore-* files and no canonical dump. DumpOneTo takes the final path and derives its own .tmp from it. DumpOne keeps its signature and calls it with the canonical name. writeSafetyDump asks for its own name directly; the rename is gone; the comment now states the invariant and how it is enforced. db_dumps no longer lists the undo copies. All three consumers of Manifest.DBDumps were grepped and named - all inside recovery_unit.go, none reads it for recovery. The files are neither deleted nor hidden. Tests 1485 -> 1493. FIVE red-proofs, TWO PASSED first time and both are reported: the behavioural tests inject the dump seam so a mutation inside DumpOneTo was invisible, and 1.3 had no test at all. Guards added at the layer each defect lives in; both mutations then convicted.	2026-08-22 23:39:59 +02:00
admin	1a2405e86f	R-379: --clear-restore-hold now states the required restart ... It runs as a second process: it clears settings.json but the running controller keeps its in-memory copy and goes on refusing. Measured on demo-hp - clear succeeded, file correct, start button still refused until a restart. Also records the lost-update window between the two processes, and why clearing through the running controller (the right shape) needs an operator tier the controller's HTTP surface does not have.	2026-08-22 18:21:47 +02:00
admin	2c724c9283	R-379/R-380: put the customer's undo copy back when a database restore fails ... R-379 and R-380 were one failure. Both ended with a half-restored database; the only difference was whether it looked broken. Postgres emptied and crash-looped; MariaDB applied part of the dump and reported health=healthy with a zero-row schema-version table. Measured live on demo-hp 2026-08-22. The undo copy was already taken and already good - proven by hand that day on both engines. Nothing in the product could apply it. Now it does, with the same ImportDump call, before any restart and inside the DB-only window. The WHOLE undo set, matched on this run's stamp. writeSafetyDump returned one path for an app with two databases; a rollback on that would restore one and leave the other half-written. When the rollback also fails the app is HELD STOPPED (operator ruling): a running app on a half-written database lets the customer make the damage permanent. Every start path refuses it - customer button, appstop Recover, boot sweep - via the shared driveStartGate, checked ABOVE its driveless early return because these apps have no drive. The marker is ended so nothing auto-restarts it. The row goes red. Cleared with --clear-restore-hold, an operator CLI route. --single-transaction is a belt on Postgres only; MariaDB DDL is not transactional and that is why the rollback is the fix. R-381: the engine's stderr stops reaching the customer (615 bytes on MariaDB, its middle rows out of their own database) and starts reaching the operator log, which never had it. R-382: the summary log prints the volume count it already held. Undo copies resolve to their own app, are marked IsUndo, and are capped at 3 per app, pruned from the capture side. The reported render-as-an-app symptom did NOT reproduce - the live page was read first and had zero occurrences. Tests 1468 -> 1483. Eight red-proofs; ONE PASSED and is reported: the R-381 behavioural test injected below ImportDump. A guard at that layer now convicts.	2026-08-22 18:03:18 +02:00
admin	08eb1a6e3a	R-356: the off-site restore refused every app that has no data drive ... ReconstituteFromOffsite and PlaceOffsiteRestore both resolved the restore destination with the RAW HDD_PATH and read an empty answer as "the app is not installed". For 40 of the 53 catalogue apps that answer is correctly empty and permanent, so both actions refused forever for a running, healthy app — and told the customer to reinstall it "in the same place", which those apps never offer. Separate the two questions. "Installed?" is asked of ListDeployedStacks via a new Manager.isStackDeployed that fails CLOSED on a nil provider. "Where?" is answered by GetAppDrivePath — the same resolver CaptureRecoveryUnit wrote the snapshot with, so the restore aims at the place the backup came from. The 13 drive apps are unchanged: own drive, mismatch check, ack still required. A third refusal, with its own sentence, covers installed-but-no-resolvable-root. Fixtures that marked an app "installed" by giving it an HDD path now state deployment as its own fact. No assertion weakened.	2026-08-22 13:08:46 +02:00
admin	34d83f5a02	feat(disk-health): poll hourly, not 6-hourly — measured, not assumed ... Part 4 was gated on a measurement. On demo-hp (Tier 0) the controller's real /disks fetch — fetchDisks, the same path the check uses, not the 60s card cache — costs min 0.805s / median 0.821s / max 0.841s over 10 calls, all HTTP 200, across 3 physical disk rows (2 distinct devices). Median is 6x under the 5s bar, so the <5s branch applies and the interval drops 6h -> 1h. Why it matters: the one real failing drive's benign excursion lasted about ONE HOUR and cleared completely. A 6-hourly sampler can land either side of an excursion like that, see nothing, and then catch the terminal run half a day late. The smartd history that produced the whole analysis sampled every 30 minutes and only just resolved the shape.	2026-08-14 08:21:55 +02:00
admin	72368654e4	R-241 part 5: escalating reminders, and operator levers for a running countdown ... REMINDERS (SEC 2.3). The offer epoch now stamps when it began, and the undecided reminder escalates in EMPHASIS at 1, 3, 7 and 14 days. THE READING IS STATED BECAUSE THE SPEC IS AMBIGUOUS, and it is written into the code where it can be corrected. For an ABANDONING box, 5/3/1 are unambiguously days REMAINING before a deletion. An undecided box has no deadline - nothing counts down to anything, because SEC 7.5 deliberately does NOT auto-abandon - so 14/7/3/1 cannot be "remaining" and are taken as days ELAPSED, with the wording firming up rather than the bar appearing and disappearing. If the operator meant something else, one function changes. The stamp is re-set on every entry into the offered state, so a box that settles and is later rebuilt starts its ladder again instead of inheriting an old one. OPERATOR LEVERS (SEC 7.5). --abandon-status, --abandon-extend=N and --abandon-stop on the controller CLI, beside the existing operator subcommands. They exist because the path that ACTUALLY happens is the customer telephoning, and support needs something to press. They live on the CLI and not in the customer UI deliberately: extending a deletion the customer asked for is an operator judgement, and a customer who wants it stopped already has the self-service route - they recover with their code, which cancels it. BOTH REFUSE RATHER THAN NO-OP, in two situations: when no countdown is running, and when the store has already been deleted. A silent success is the thing an operator most easily mistakes for "handled" - they would tell the customer their data was safe when it is gone. Pinned by two tests. --abandon-extend counts from NOW, not from the old due date, and a test proves the old date passes without deleting anything. Green: go build, go vet, go test ./... all pass; controller gates OK.	2026-08-07 12:08:11 +02:00
admin	a5d90ff801	R-241 part 3: abandoning starts a 14-day countdown that ends the question ... Until now "set aside" renamed the remote store and touched neither the escrow nor the key, so the hub went on holding a sealed package for a key the box no longer used. Shape (c) compares those two, finds them different, and offers recovery - correctly, and for ever. A customer who had already said "I do not want the old data" would be asked again at every login. The operator's ruling is that the answer is NOT a "they decided" flag: fix the state, do not remember that it is wrong. So the decision starts a countdown, at the end of which the set-aside store and the sealed package that protects it are removed TOGETHER. Afterwards shape (c) has nothing to compare and the offer falls silent on its own - because the state is right, not because something remembers it once was not. THE GRACE IS REAL. The recovery offer stays reachable for the whole 14 days; that is the change-of-mind path, and a grace in which recovery is impossible would be decorative. BOTH HALVES OR NEITHER. Removing only the store leaves a package that opens nothing; removing only the package leaves ciphertext nobody can ever decrypt. The two cannot be atomic across two machines, so it is a two-phase commit: delete the store, record a durable marker, and keep DECLARING offsite.abandon_purge_requested until the hub's ACK stops reporting a superseded package. A crash between the halves re-declares on the next sweep; it never leaves the pair half-removed and silent. HUB HALF - SEC 8.2 ANSWERED: yes, the hub was needed, and only for this. store.PurgeSupersededEscrowForCustomer is the one place R-198's retention is ever undone, and it never touches host_escrow (the package covering the key the box uses now). The handler acts on the DECLARATION, never an inference, and is placed immediately BEFORE the ACK is built - so GetEscrowStatusForCustomer reads the effect and the SAME response closes the box's two-phase commit. No second round-trip and no window where the box thinks it is still owed. felhom-agent was NOT touched. The countdown starts in ResetOrphanedRepo, NOT in the shared helper: the helper is also the unclaimed auto-reset path, where nobody decided anything, and an as-delivered box tidying a stranger's leftover store must not get a customer's deletion clock. Pinned by a test. Cancellation is wired into the recovery unlock, BEFORE the tier-up and the listing - those can fail, and a countdown surviving a successful unlock because a later step errored would delete the history the customer just proved they can open. The sweep is a Daily job at 05:10, not on the backup leg: it must run on a box whose tier is not configured for runs. Quiet by construction on every box with no countdown, and that silence is asserted. Tests (all clock-injected; SEC 7.4 forbids shortening a live timer): Scenario E (aside + package kept + countdown + offer still reachable, and NOTHING deleted), Scenario F (both halves, the declaration repeating, the close-out), Scenario G (cancel, path still nameable, no later deletion), plus: not closed out while the package remains, a transport failure leaves the countdown due and retrying, the no-op sweep issues zero remote commands, and the unclaimed auto-reset starts no countdown. RED-PROOFS, each with the mutation confirmed present in the file first: F1) store deletion skipped -> Scenario F FAILS (no rm issued) F2) declaration dropped from the report -> Scenario F FAILS (the hub is never asked; the package would outlive the store for ever) G) CancelAbandon made a no-op -> Scenario G FAILS (uncancellable countdown) Green: controller and hub both build, vet and test clean; controller gates OK. NOTHING WAS DELETED ANYWHERE - the terminal step has only ever run against in-test fakes.	2026-08-07 11:47:42 +02:00
admin	a491abef6c	R-241 part 2: the comparison the box already makes becomes the thing that offers recovery ... THE FACT WAS COMPUTED EVERY CYCLE AND KEPT NOWHERE. EscrowAutoConfirmer.Reconcile has compared the hub's restic_pw_sha256 against the local key on every ACK since SLICE 3. On the final-walk venue it logged, at 03:28:03Z and thirty-five minutes before the customer looked, "the hub's escrow blob does not cover the CURRENT repo password (hub hash 30ef574f != local 9b4a9a9d)" - and dropped it. The recovery screen, evaluating in the same process, went on asking a question that could not see it. Now persisted: settings.HubEscrowKeySHA256 + HubEscrowKeyCheckedAt, recorded UNCONDITIONALLY in Reconcile beside RecordPresence and RecordSuperseded - same place, same reason: the box that needs it most is the rebuilt one with no target, on which every gate below returns early. OffsiteRecoveryOffer gains SHAPE (c): the hub holds a package for a key OTHER than the one we are using. (a) and (b) are both proxies for that question and both have now been wrong in opposite directions - (a) goes false the moment anything mints, (b) is unreachable while the escrow is pending. SEC 7.2, decided deliberately and stated in the code: - a KNOWN DIFFERENCE offers, however old the reading. Age is not gated on. Both sides are local; only the hub's half can be stale, and what the hub holds does not change without a ceremony THIS box runs, which refreshes the hash on the next ACK. Gating on age would make a box offline from the hub silently stop offering - the exact failure this session removes. CheckedAt is persisted for diagnosis, not as a gate. - an ABSENT hash falls back to (a)/(b) and does NOT offer. "" is the hub positively saying its package seals no repository password (legacy hash-less escrow). Nothing to compare, and offering would put a permanent screen in front of every legacy box. The write damper: CheckedAt refreshes on every ack carrying a hash, but a save is skipped when both the hash and the UTC day are unchanged, so an idle box does not rewrite settings.json every fifteen minutes. It records WHEN WE LAST HEARD, not when it last changed - the R-100 distinction. Tests: Scenario C (a differing key offers, with both proxies asserted false first), Scenario D (a matching key offers nothing), fact 1 still required, shape (a) still works, and both SEC 7.2 halves. RED-PROOFS, each with the mutation confirmed present in the file first: D) hubHash != localHash conjunct dropped -> Scenario D FAILS (a healthy box offered recovery forever); Scenario C still passes WIRING) RecordEscrowKeyHash removed from the EscrowAutoConfirmer literal in main.go -> TestMainWiresRecordEscrowKeyHash FAILS. This is the ships-inert shape: unwired, everything compiles, every test in the package passes, the auto-confirm still works, and shape (c) reads an empty hash forever. Green: go build, go vet, go test ./... all pass.	2026-08-07 11:33:42 +02:00
admin	9dc26459ea	v0.203.0: the box collects what the hub staged for it (R-218 consume half) + R-220's message ... R-218's declaration half shipped in v0.201.0 and works. Its consume half never existed. Reconcile ran exactly twice per process — at start-up and when the recovery screen drives it — and BOTH fire before the hub has anything staged, because the hub stages in RESPONSE to the declaration those runs precede. Measured on the R-201 re-walk: unlock reconcile 11:43:07, hub staged 11:44:57 saying 'next cycle', a full report cycle ran 11:55:46, still unconsumed at 12:06. A guest command line applied it in 18 seconds — everything correct except the trigger. Bridge.RetryIfDeclared re-runs the SAME reconcile on a 5-minute tick, driven from the box's own published declaration (OffboxReportStatus().State) — the very statement the hub acts on, so the two cannot disagree. Poll, not an ACK flag, decided on the promise: the no-target message says 'amint megvannak' (no deadline) and the card says 'within a day'. Five minutes is inside both by a wide margin and needs no hub change. It stops by construction — a healthy box does no work and logs nothing — and the settle gate is deliberately kept via ReconcileWhenSettled. The marker was investigated and left alone: applied_marker lives in the guest's DataDir, which a rebuild destroys, so it cannot suppress a legitimate re-run. R-220's customer half: the refusal no longer tells the customer to choose from a list that may be empty. It names the rebuild, points at the Meghajtók page, and promises no outcome. Red-proofs: remove the retry -> credential uncollected (the dead end reproduced); drop the stop condition -> a healthy box hammers the hub; call Reconcile instead of ReconcileWhenSettled -> settle gate bypassed; restore the old sentence -> the impossible action returns. 28 packages ok, vet clean, all controller gates OK.	2026-08-06 12:56:12 +02:00
admin	a3499d1807	v0.201.0 — a correct recovery code is never called wrong again (CAMPAIGN-11) — MinAgent 0.125.0 ... R-216: the offsite key recovery is a coupled feature and now says so. featureProbes + featureMinAgent 0.125.0 + a Supports gate at the unlock entry point, FAILING CLOSED — an agent that cannot answer is named as such instead of the customer's code being blamed. Measured live: a 404 from agent 0.120.0 came back as "we did not accept your recovery code, check that all ten words", in 0.134 s, against a perfect code. R-218: delete the repo-password short-circuit in needsOffsiteCredential. The declaration stops when the TIER WORKS, not when a key exists — installing a key is the recovery screen's whole job, so succeeding at recovery was switching off the mechanism that would have delivered the coordinates to use it. R-219: the unlock finishes the job — place the key, bring the tier up, then list. Without it the promised listing could never render on the shape the screen exists for. R-217: an unreadable store no longer claims to have opened with unattributable content (the OffsiteInventory{} zero value). Opened / empty / unreadable are three states. R-222: a code that is right about a RETAINED earlier package is named, not blamed. States what the hub knows and promises nothing — no read path exists. R-215: GET /recovery is gated on the same predicate as the interception. Five red-proofs, each demonstrated failing and restored.	2026-08-05 17:48:08 +02:00
admin	636c51e542	R-193: the recovery screen — unlocking, and only unlocking (v0.200.0) ... A customer whose machine was rebuilt had everything needed to get their data back and no way to find out: the only route was a command line. This is the screen that closes that. IT UNLOCKS, AND ONLY UNLOCKS (operator ruling). It explains, takes the recovery code, opens the repository and shows what is in there — apps, dates, sizes. It restores nothing: restore is already per-app and lives in the backups area, and a screen that unlocks and then offers to overwrite is two decisions wearing one button. ONE CORE, TWO CALLERS. RecoverInstallCore is split out of RecoverAndInstall; the CLI wrapper keeps its exit codes and printed lines byte-identical, and the handler drives the same function. Two implementations of the one operation that can permanently lose a customer's data would drift, and only one would be tested. Asserted from source on both sides by AST. THREE WAYS OUT, none a dismiss button: recover; 'most nem' (the full page stops interrupting, the backups-area entry point stays PERMANENTLY, bound to the offer and never to the postpone flag); and 'I do not want the old data' — confirmed TWICE and reaching the SHIPPED move-aside, which sets aside and never deletes. THE CODE IS HANDLED NO MORE LOOSELY THAN ON THE COMMAND LINE: POST body only, never logged, never persisted, never echoed, cleared on every path, no-store, autocomplete off. No lockout — the code is a ten-word phrase, and locking a customer out of their own data for a typo is worse than anything it prevents. TWO DEFECTS THE TESTS CAUGHT, both fixed: an UNCLAIMED (legacy-open) box would have been shown the page, because RequireAuth passes such a box through; and the inventory nil-dereferenced when no off-site target was configured, which is exactly the pristine rebuilt shape.	2026-08-05 12:45:48 +02:00
admin	1214bae0a2	R-204 item 4 (box half): a rebuilt box DECLARES that it needs a credential (v0.199.0) ... An absent off-site object has four meanings — never configured, mid-restart, a transient config read failure, and rebuilt-and-stranded — and the hub cannot tell them apart. The box can, from two local facts it holds with certainty, so it says so instead of leaving the hub to deduce it from a silence (operator ruling). The ACK's identity_blob_present is now recorded on EVERY ACK, before the gates that used to discard it: on a box with no off-site target the auto-confirm returns immediately, which is exactly a rebuilt box, so the one fact distinguishing it from a box that never had off-site backups was thrown away every cycle. The declaration needs BOTH halves — a fresh data area AND a hub-held recovery package. Freshness alone is a box that never had off-site backups; dropping that condition makes the whole fleet ask for credentials, which is what the Scenario B test exists to catch. The object carries enabled:false and zero sizes, which is what makes it inert to the hub's existing fill and staleness checkers and to a pre-upgrade hub. A configured box's JSON is byte-identical to v0.198.0's.	2026-08-05 10:47:51 +02:00
admin	58c703bd44	R-203 Part 2: a run that missed a MANDATORY directory is not a successful run (v0.197.0) ... The gap was already detected and warned about, in Hungarian, naming the app and the folders -- that warning is what stopped the R-201 drill. The defect was that the run still reported `ok` beside it, and a warning standing beside a success is read as a success. last_status gains "incomplete": minted, because "ok" | "error" | "running" had nothing meaning "it ran, and this app is not fully protected". NOT "error" -- the rest of the run worked and what was captured is real, so SnapshotCount and the LastSuccess anchor still record it. Half a backup is not no backup. The gaps are now recorded STRUCTURALLY (offboxRunResult.mandatoryGaps), not only as prose, so the verdict has something to act on. It reaches the operator through the EXISTING per-run digest (backup_run_failures) rather than a new event type -- a new type is a two-repo change and the hub drops anything outside allowedEventTypes. The stat-filter gains the ClassMandatory check Tier 2 already had. It is a NO-OP today (TierOffsite admits mandatory only), so no customer-visible warning disappears -- demonstrated by widening the tier filter alone and watching the check hold the line. ANTICIPATED: calibre-web on demo-hp has exactly this gap, so its off-site status becomes incomplete the moment this ships. That is correct and is the point. Red-proofs: my first Scenario-C proof PASSED because the test only reached offboxCaptureSet while the mutation lives in runOffboxInternal -- a mutation the test cannot observe is not a red-proof, and the fix was the test. The run-level test now fails under both mutations (unreachable gap recording; unconditional ok).	2026-08-04 18:32:56 +02:00
admin	a96c3d9473	R-203: the export-mount resolver takes the namespace root too (its own commit) ... ExportDataMounts lives in delete.go, which reads as a destructive path. IT IS NOT: its single production caller is the .fab export adapter, and nothing deletes based on its result. The delete path's own guard, ProtectedHDDPaths, is layout-agnostic by construction -- it protects BOTH <hdd>/... and <hdd>/felhom-data/... -- so deletion was never affected by the namespace-root defect. That scope note is now in the function's doc comment, because the file placement will mislead the next reader exactly as it misled the spec for this change. Separated into its own commit anyway, so a change to a function whose filename says "delete" is reviewable on its own. An empty nsRoot falls back to hddPath -- the pre-R-203 shape -- so any caller not yet updated keeps working on enrolled drives. Tests cover both drive kinds and assert the NEGATIVE: no emitted path lies outside the app's own data roots. Red-proof: leaving the site bare fails the system-drive row, emitting /mnt/sys_drive/userdata where the canonical root is /mnt/sys_drive/felhom-data/userdata.	2026-08-04 18:21:17 +02:00
admin	73efb091d9	R-203: the app and its backup look in the same directory — one resolver, every caller ... appbackup's path helpers take a NAMESPACE ROOT. Five call sites passed a bare DRIVE path. On an enrolled drive the two coincide, so nothing showed; on the system-data fallback they differ by exactly the felhom-data segment, and the app then bound a directory the off-site capture set never looked at -- while the run reported ok. Measured live on demo-hp: the app wrote to /mnt/sys_drive/userdata/media/books, the capture set looked for /mnt/sys_drive/felhom-data/userdata/media/books. THE RULE NOW HAS ONE EXPRESSION. appbackup.NamespaceRootFor / IsEnrolledDrive encode the drive-kind comparison; backup.Manager.namespaceRoot and stacks.Manager.inGuest delegate to it. There were already TWO copies and they differed -- the backup package's compared without filepath.Clean, the stacks package's with it, so a trailing slash from config would have flipped the mode in one and not the other. Sites routed through it: - stacks/deploy.go withPathVars -> ${USERDATA_PATH} (the live defect) - appexport/fabplan.go + export.go (via a new provider method) - web/handlers.go FileBrowser mounts (latent: the system drive is deliberately never a registered StoragePath, so this is the identity today) ComputeFabBuckets now receives the namespace root, which is what ComputeCaptureSet has always received -- so the export's classified paths and the backup's capture set describe the same directories by construction instead of by coincidence. Tests are table-driven over BOTH drive kinds, because this survived by being invisible on the kind that already worked. Red-proofs observed: restoring the bare-path call fails the system-drive row with the two paths differing by /felhom-data; inverting the drive-kind comparison fails every enrolled row.	2026-08-04 18:17:05 +02:00
admin	1b1366bb6e	controller v0.196.0: the recovered key installs itself (R-200 plumbing half) -- MinAgent 0.125.0 ... --recover-offsite-install is the sibling of --recover-offsite-check: same fetch/unseal path through the agent, same STDIN discipline for R, but it PLACES the recovered repository password via InjectOffboxPassword so a rebuilt box reopens the history it inherited. Doing this by hand would put the offsite DATA key through a terminal, a clipboard and shell history. In-process the value goes agent -> this process -> the 0600 file and is rendered nowhere. The confirmation is a SECOND invocation: without --confirm-install it prints both hashes and writes nothing, so the operator sees the comparison before any write is possible. Three outcomes, named distinctly: installed (no local password -- the rebuilt-box shape), unchanged (identical key already present, nothing written), refused (a DIFFERENT key present; installing would clobber the key the current repository is encrypted under, and no force option is offered). Exit 2 for the refusal, distinct from 1 for a failed step. Red-proof: removing the confirmation gate makes the dry run write, failing the test. The R-persistence test carries a positive control -- a planted copy is found, then removed and not found -- because an absence check is worth only what its sensitivity is.	2026-08-04 14:27:38 +02:00
admin	9640e51321	controller v0.195.0: prove the offsite key comes back (R-200 plumbing half) -- MinAgent 0.125.0 ... --recover-offsite-check is a docker exec diagnostic in the shape of --print-reset-code: it reads the customer's recovery code from STDIN, asks the agent to fetch this host's sealed bundle and open it, and reports whether the recovered key matches the one on disk BY SHA256. Two hashes and a verdict; never a password, never R, never a blob. R comes from stdin and not a flag because a flag value is visible in ps, in shell history, in a container's command line and in any transcript of the session that ran it. IT COMPARES; IT DOES NOT INSTALL. The recovered password is never written to offbox/repo_password -- installing changes a live box on a path nobody has walked, and that link is next session's, with the drill around it. A test asserts the data dir is byte-unchanged after a check; its red-proof (adding the install call) fails it. Exit codes: 0 match, 2 clean MISMATCH, 1 a step failed -- "it failed" and "it worked and disagreed" must never share a status. A box with no local password reports distinctly: that is the rebuilt-box shape, where the next step is to install rather than compare. Nothing customer-reachable ships here: no card, no form, no preview.	2026-08-04 13:42:50 +02:00
admin	88897a224e	v0.194.0 — one operator email per backup run, and nothing dropped without a trace (R-182) ... MEASURED, not supposed. On 2026-08-03 nine per-app recovery_unit_capture_failed events reached the hub and TWO operator emails went out. The hub's operator cooldown key is customerID:eventType(+tier) and that event carries `app` but no `tier`, so the key held no app identifier: the first refused app took the hour's slot and every other app's failure was discarded BEFORE anything was written down, leaving no row on any channel. The obvious fix — put `app` in the key — was ruled against: on a full disk it produces one email per app, the volume problem wearing the correctness problem's clothes. internal/backup/runsummary.go: a per-run collector with exactly admissionSet's lifetime, fed by all three write legs, emitting backup_run_failures ONCE at the end and only when something failed. A clean run emits nothing. The per-app event stays and becomes the RECORD — the hub routes it record-only, stored and logged every time, never competing for an email slot. The record and the notification are now different things. Deliberate skips (disconnected, decommissioned) are excluded: they have their own alert, and a nightly email about an unplugged drive is one the operator learns to ignore. A manual run always reports: the digest carries a unique run_id the cooldown cannot collapse. Someone pressing the button is actively trying to get a backup. THE PERIODIC SWEEP GETS A DIGEST TOO. With the per-app event now record-only, a capture failure found between runs would be recorded and never notified — a new silence introduced while closing one. That path emits a digest with NO run_id, so the ordinary 1-hour cooldown caps it exactly as before while the mail now lists every failing app instead of whichever was first. A refusal is recorded ONCE, where the verdict is taken, not at the three legs that consult it — R-181's contract is one verdict per app per run. Noting it per leg listed one refused app three times and produced "2 of 1 apps failed". Found by the digest's own test, not in review. Silence is safe because the hub's deadline check raises expected_backup_missed from report freshness, independently of any mail this box sends (monitor/deadline.go:396,417). Confirmed, not assumed. 7 new tests, 4 red-proofs. The main.go seam walk did NOT fail on its first attempt — the AST test walked the backup package and not main.go; the test was fixed and the mutation re-run rather than the pass recorded.	2026-08-03 13:46:14 +02:00
admin	5adae4dad9	v0.191.1 — the fill check also runs at startup (R-167) ... Found live on 9201: neither sched.Daily nor sched.Every fires on registration, so a box booting with a filesystem already over the line would stay silent for up to 24h — the R-100 shape, and the same gap the hub's own checkers avoid by leaving already-breached keys unseeded at init. The watcher now runs once 90s after startup as well. Safe because the check is edge-triggered against persisted state: an already-warned filesystem stays silent. The delay lets mounts settle so a drive still returning reads as unreadable and is skipped rather than warned about. Pinned by an AST assertion — the schedule registration alone no longer satisfies the test.	2026-08-02 23:27:35 +02:00
admin	cf48214f6c	v0.191.0 — warn before the wall comes down (R-167, R-158, R-174) ... R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills. It emits the PRE-EXISTING disk_warning/disk_critical pair, which was allowlisted, copy'd, default-enabled and checkbox'd with no producer in any repo — the sixth "built but never wired" instance here. Two threshold terms (85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at both ends of this fleet's size range. Edge-triggered on escalation only, state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil usage read is never a warning and never clears one. Per filesystem, never per app. Daily 03:30, before the nightly app-data legs. R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture fails, loop continuing, carrying the target filesystem's used/free bytes. Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed, which is customer-enabled and would email the customer about a failure they cannot act on. D-c overrides R-158's own proposal here. R-174: the app-stop guard no longer starts apps onto MISSING drives — a regression in v0.189.0 code, found by review and closed the same session. SetStarter got the raw stack manager, whose StartStack has no drive gate, and Recover runs at startup. R-171 one path over. bootDriveGate could not be reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read vars assigned after Recover runs), so holder #3 is extracted into a shared driveStartGate with a test pinning the delegation. ErrStartRefused splits a refusal from a failure: both keep the marker, only Failed alarms, because routing a deliberate hold into NotifyBackupFailed is the same false alarm. Tests 1157 -> 1184. All red-proofs demonstrated failing and restored.	2026-08-02 23:18:51 +02:00
admin	dcc3363d2f	boot window: sample REFRESHES first — a cached fleet made 'settled' meaningless ... Found by live validation on 9201, not by review. GetStacks() is the Manager's in-memory map refreshed by the scheduler every 10s; sampling it every 5s without refreshing means two identical samples can mean the cache did not update rather than that the fleet settled. A container removed ~5s before the window closed was still in the sampled fleet and the sweep logged 'no boot-orphaned apps' for an app that had none. sampleBootFleet now refreshes first; a refresh error degrades rather than aborting the window.	2026-08-02 20:17:12 +02:00
admin	582135f861	v0.190.0 — the boot settle window, both gates on intent, and R-171 ... R-171 (a regression v0.189.0 introduced, CONFIRMED on hardware before any fix was written). Replacing isBootOrphan's container-count term with recorded intent made a drive-gate-stopped app read as a boot orphan: the gate stops apps with `compose down` (zero containers) and never touches desired_state, because it is not the customer. Observed on 9201 with the drive held unmounted — the sweep found and started it, burned both attempts, and handed it to the dead-app alarm. The write hazard did not materialise (the unbound mountpoint is host-root-owned and the guest is unprivileged) but that protection is accidental and untested. New consumer-side seam bootrecon.StartGate, fail-safe (cannot determine ⇒ do not start), wired in main.go. The rule is not new: the API's startGatedByMissingDrive already refuses this; the sweep bypassed it. R-157 mechanism A. The sweep looked once at T+5s, deriving candidates from a fleet docker was still restoring — three of six hard resets. Now a settle-then- sweep window: sample every 5s, settled after 3 identical samples, sweep ONCE at the end; ends on settled or a 50s budget, and the log says which. The budget is 50s because settle+budget+one retry must stay under the 90s dead-app grace — a test rejected 60s at 95s. A window that overruns emits a LATE RECOVERY warn rather than the grace being widened to hide it. Widening the window made two more holders reachable, so the one gate covers all three: an absent drive, a quiesce, and an in-flight app-data operation — reusing quiesce.SuppressedStacks() and a new read-only AppStopGuard.HeldStacks(). R-170. shouldRecreateOnBoot now reads desired_state with the identical three-way table; absent keeps the old hasContainers behaviour exactly. Its comment argued for the container count and was rewritten. presentStable is untouched. The two gates' agreement is pinned from both sides against one fixture table. 27/27 packages green; 6 red-proofs observed FAIL then restored.	2026-08-02 19:56:20 +02:00
admin	dbcb306fcf	v0.189.0 — desired state + the app-stop crash marker (R-166 / D-b) ... The box stops inferring the customer's intent from a container count and reads what they actually asked for. Part 1 — desired state. AppConfig gains a tri-state `desired_state` (""/running/stopped), written ONLY by the customer's own action: the API action switch, DeployStack, UpdateOptionalConfig's redeploy branch, and the .fab import. Intent is written BEFORE the act and a failed write REFUSES the act. StartStack/StopStack are deliberately not writers — 14 callers, only 2 are the customer. bootrecon.isBootOrphan now reads intent instead of len(Containers)>0, which closes R-157 mechanism B (a power cut or interrupted deploy left an app with zero containers, read as a deliberate stop, and stranded silently). ABSENT MEANS UNKNOWN, NEVER "running": every pre-v0.189.0 app.yaml reads absent, so the legacy fallback is byte-identical to the old rule. A running-only startup backfill converges the unambiguous cases; `stopped` is never inferred. Part 2 — backup.AppStopGuard, a persisted marker over every stop→work→start window (volume dump, offbox reconstitute, .fab export). Its own file, never quiesce's. Written before the stop, cleared only after a restart that succeeded, kept when one fails. Recover() completes before the boot reconciler is launched and returns its outcome, which main.go reports on the existing backup_failed event once the notifier exists. A defer is not the mechanism — a SIGKILL runs none (Campaign 8 fault 10). Also: SaveAppConfig rebuilt AppConfig field-by-field (the R-100 shape) and would have dropped desired_state on every save across nine call sites. Replaced with copy-and-overlay. Measured: app.yaml does not round-trip unknown YAML keys. No hub change, no agent coupling, no user-visible string. 27/27 packages green; 7 red-proofs observed FAIL then restored.	2026-08-02 18:40:17 +02:00
admin	4ed938cce4	D5: an app restore works from the drive alone (v0.188.0) ... The recovery unit on the customer's drive now carries the PORTABLE secret class, so Tier-1/Tier-2 restore no longer depends on the whole-guest tier. A customer needs the drive and nothing else. Part 0's rulings overturned the brief's recommendation, on evidence: - the data_key flag is untrustworthy (4+ encryption keys the catalog itself labels as such are unflagged) -> R-127 - a DB password is not resettable in practice: POSTGRES_PASSWORD is ignored once PGDATA is non-empty, so a regenerated value leaves the app unable to authenticate against its own restored rows while the dump replay still reports success (proven on a throwaway postgres:16-alpine) Ruling (operator): type:secret travels, type:password never does, minus the nonPortableSecrets code register. Plaintext -- withholding the internet- reachable class is what licenses that, and the two are coupled. Precedence: the UNIT WINS over the guest -- the unit's secrets were captured in the same run as the dumps beside them, so they match the data being restored. The fail-closed data-key gate is unchanged. Secret values are never logged; the manifest records NAMES only.	2026-07-30 16:33:06 +02:00
admin	fd50a73e65	C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (v0.183.0) ... Both are the system reporting healthy while the customer is not, and both live in the same status-derivation code. Neither is fixed by making the system quieter. C9-F1 (HIGH) — Tier-2 writes recovery-unit/ on EVERY run and RestoreTier2Files has never read it (tier2_restore.go:101-104 reads hdd/ + userdata/ only). Phase 0 enumerated all 53 catalog templates against both demo boxes: 43 apps have NO readable subtree, so the button stopped the app, restored 0 files, restarted it and said "Nincs hiányzó fájl — minden fájl megvan a helyén." — at the moment the customer pressed it because files were missing, with 156 MB of BookStack's data unread in the same copy. 9 apps have file legs but never their DB or volumes, so the same sentence was also a clean bill of health over data never opened (immich: 1.3 GB Postgres unit). Honesty half shipped: a pre-flight coverage check refuses UP FRONT without stopping the app and NAMES the action that works; a run that proceeds claims only what it EXAMINED and discloses that the database and volumes are not covered. Completeness is filed as C9-F1b — routing to the Tier-1 unit restore puts a destructive operation behind a non-destructive button, so its confirm copy has to carry that difference. C9-F4 filed: nothing reads the Tier-2 recovery-unit/ mirror, so the second local copy that exists for drive loss is unreachable by any customer action. C9-F2 (HIGH) — a crash loop was counted as working. StateRestarting is deliberately NOT added to IsDownState (that alarms on every deploy fleet-wide, the over-correction F-A1 nearly cost us); a sustained run becomes down after crashLoopAfter = 5m, set above the 120s deploy timeout, Mealie's 60s start_period and R-97b's 180s grace. The dashboard counter uses the same predicate, so it no longer contradicts the alarm on the same screen. README's claim that faults "still surface as restarting" was a wish with no test — corrected in place; it is the seventh such instance. Six red-proofs observed, including the one that matters most: adding StateRestarting to IsDownState fails the brief-restart test with "every deploy and update would page the operator". go test ./... rc=0, 27 packages, run and read separately from this commit.	2026-07-28 18:53:56 +02:00
admin	3f048e042b	R-101 + F-DIAG: the restore dialog names the last SUCCESSFUL copy (v0.182.0) ... Tier2LastRun is the attempt clock and was rendered as 'Legutóbbi másolat' in the restore confirm dialog. New LastSuccess + SuccessTracked anchor; tier2Update makes the three rebuild sites safe by construction. F-DIAG: six distinct causes, target-aware redaction.	2026-07-28 16:36:47 +02:00
admin	fb91c8d766	F-OBS: the dead-app check gets a positive observable (v0.180.0) ... deadapp-check had no observable at default info level: its per-cycle line goes through Scheduler.dbg(), gated on logging.level==debug, so on a default box it is never PRODUCED (not merely filtered) and cannot reach the always-DEBUG ring. A 30s interval also puts it on the scheduler's quiet path. 'No alarms' was therefore indistinguishable from 'the detector never ran' — which undermines confidence in the F-CRIT-1 fix in the field. A periodic summary, not a line per run: at 30s a per-run line is 2880 lines/day, which is why the original author chose silence. Every 20th scan (~10 min) emits one INFO with the scan count, apps evaluated and apps down. A test pins the cadence so it cannot be widened into uselessness. Also corrects the 'unquiesce guaranteed by defer' comment — fault 10 established the guarantee is the crash marker plus Recover().	2026-07-28 10:27:28 +02:00
admin	079265ad8e	F-CRIT-1 + F-A1: one alarm that never fired, one that fired wrongly (v0.179.0) ... F-CRIT-1 — an app that failed to restart after a quiesce never alarmed, for two independent reasons, either of which alone kept it dead: 1. restartAll returned nothing, so the failure was logged and dropped and no caller could learn a customer's app had not come back. It now returns the stacks that failed; both call sites record the outcome. 2. classifyRunStates whitelists StateStopped on invariant I1 ('StateStopped means the user stopped it'). The quiesce loop stops stacks by the same compose-down path, so a failed restart is also StateStopped and was whitelisted into silence. Loop.FailedRestarts() is now the only thing that lifts the whitelist, so genuine user stops stay silent (v0.164.0 pinned). F-A1 — HTTP 409 is the agent's single-flight gate refusing while a restore-test holds it, not a failure. agentapi now returns a typed *StatusError on POST, the adapter maps 409 -> quiesce.ErrTierBusy, and the loop defers: no breaker, no event, no operator email, tier stays DUE. Two traps avoided. Silence: contention outliving contentionAlarmAfter (3h, set by the agent's own 120m PBS restore-test ceiling) raises its own BLOCKED signal. App thrash: removing the failure treatment also removes the breaker's deferral, so a contended tier is dropped BEFORE anything stops (contentionRetryAfter 15m, against a 12m01s longest observed restore-test). Three comments corrected; the invariant rule added to both CLAUDE.md copies. Six red-proofs, all observed failing.	2026-07-28 08:50:11 +02:00
admin	8f46495426	seam sweep: move the TieredBackend witness into production code (no version bump) ... A witness in a _test.go file fires on go test and go vet but NOT on go build alone — and a build-only step is exactly how the R-88 Part 2 near-miss would have shipped. Moved beside the type it pins, and added one for AgentVersionReporter. No defect found: quiesceBackend and *Client both satisfy their interfaces today. No version bump, no deploy — compile-time only.	2026-07-27 18:23:50 +02:00
admin	86ea482fc1	controller v0.178.0 — R-88 Part 2: only a positive 'never' fires the valve ... MinAgent: 0.105.0. scheduledRunAllowed fired on any nil age; it now requires a licence from valveLicensed, which grants it for AgeStateAbsent and for a LEGACY agent, and refuses it for AgeStateUnknown. An unreadable storage no longer masquerades as a first-ever backup and no longer quiesces apps outside the window. A missing wire field means legacy, not unknown — deliberately. Treating it as unknown would stop the valve firing on un-upgraded boxes and starve genuinely new ones. Degrade logged once; unrecognised future values also map to legacy. Caught in passing: TieredBackend is satisfied by a RUNTIME assertion, so the signature change compiled and vetted clean while quiesceBackend silently stopped satisfying it — which would have degraded every box to the single-tier path with no error. Added a compile-time witness. Also corrects the notifier comment that claimed operator-only came from a missing customerMessages entry; enforcement is hub-side operatorOnlyEvents (hub 0.79.0).	2026-07-27 18:08:56 +02:00
admin	e9c99566b0	R-97: a failing backup is heard, and stops blaming the apps (v0.177.0) ... R-97a: internal/quiesce had no route to the hub at all — three failed whole-guest backups on 2026-07-27 produced zero events. TierNotifier is a seam (not an import), wired by an init-only setter because main.go builds the notifier after the loop. Edge-triggered: the failure fires when the R-88 breaker ARMS, not per retry, and recovery rides recordSuccess's existing bool. Uses NEW operator-only event types; reusing backup_failed would have emailed the customer in Hungarian about a backup they cannot act on, since it has a customerMessages entry and is in live enabled_events. Requires hub >= v0.78.0. R-97b: v0.164.0's state filter cannot see an app caught MID-RESTART, which is how BookStack alarmed. The fix is a suppression window keyed to the quiesce CYCLE, consumed at the same single derivation point. 180s grace, derived from the deploy flow's 120s health timeout and Mealie's 60s start_period; it expires, so an app that genuinely fails to come back still alarms.	2026-07-27 17:01:41 +02:00
Claude Code	de96efc0c5	v0.174.0 — R-82 Slice B: one quiesce window, two backup tiers ... MinAgent UNCHANGED — degrades gracefully against ANY older agent. The agent gained per-target tiers in v0.97.0. The controller owns quiescing, so the multi-tier schedule is reconciled here: every due tier is collected up front and run inside ONE quiesce window (one stop, N sequential backups, one resume). Two cycles on the weekly night would mean two app outages for one night's work. Dedup rule: local-only -> one quiesce; PBS-only -> one quiesce; BOTH due -> ONE window with both backups inside; neither -> no quiesce. - quiesce.TieredBackend + BackupTier + ErrTiersUnsupported (optional extension) - agentapi: BackupTiers/BackupDueFor/StartBackupFor/BackupStatusFor; targetQuery("") yields an EMPTY suffix so untargeted hits the pre-R-82 route byte-for-byte - Loop.resolveDueTiers = the dedup rule in one place, agent order preserved - quiesceAndPollTiers + pollTier: app stays quiesced until the LAST tier snapshots (resuming earlier loses app-consistency on the DR tier). Consequence stated in the docs: both-due-night downtime = first tier's full backup + last tier's snapshot, which is why tiers run fast-first. - Manual 'Mentes most' covers EVERY tier, due-ness ignored. - Window-gate safety valve now uses the OLDEST due tier, so a stale DR tier cannot be starved by a fresher local one. Capability detection: /backup/tiers 404 = pre-R-82 agent (the documented route-probe mechanism). Not a featureProbes row on purpose — the loop needs the tier LIST, not a yes/no. Degrade logged exactly once per process. Tests +11, full suite green. Red-proofs #2 and #3 observed and restored.	2026-07-26 14:40:44 +02:00
admin	9056f01fae	v0.173.0 — R-77: endpoint-drift detection, samba protected-set gate, channel log honesty ... Source: felhom.eu/documentation/audits/DIAG-agent-channel-2026-07-26.md bootstrap.DetectEndpointDrift names a controller.yaml vs bootstrap.json local_api.endpoint divergence -- one ERROR carrying BOTH values and BOTH paths, its own event type local_api_endpoint_drift, and its own Hungarian banner shown ABOVE the channel banner because drift is the cause and "agent unreachable" the symptom. It writes NOTHING: reconciling from bootstrap.json would clobber a correct controller.yaml on any half-provisioned or hand-repaired guest, so the authority ruling is deferred to R-78. Fail-safe silent on absent/unparseable/ incomplete bootstrap and on an empty endpoint (ensureLocalAPI's fill-if-missing path is untouched). Fingerprint compared as a BOOLEAN only; token never compared, logged or exposed. EffectiveProtected now gates samba on Enabled && UserSet, mirroring BOTH of reconcileSambaAt's early returns, and the doc comment is corrected in the same change -- it claimed "detection and deployment agree in both directions" while citing only !smb.Enabled, an assertion that went false when !smb.UserSet was added. Not over-suppressed: sharing on WITH a password and a dead container still alarms. Channel log: the debounce placeholder is stateUnconfirmed (rendered "unseeded") instead of "up", so a born-down channel no longer logs "up->down" and orUnseeded stops being dead code. Logging only -- the placeholder is still matched in the re-arm condition, so F2 born-down alerting is byte-for-byte unchanged and all nine pre-existing channelhealth tests pass. Tests 951 -> 959, all green. Red-proofs A (both directions), E and F. MinAgent unchanged; felhom-agent untouched.	2026-07-26 09:13:52 +02:00
admin	2958946517	v0.172.0 — R-75: canonical import root, catalog-derived skeleton, import surfaces ... ${IMPORT_PATH} = <system namespace root>/userdata/import — ONE drop-zone per box, on the system drive, injected at BOTH compose-env builders with NO per-drive fallback (unresolvable leaves it unset so compose fails loudly rather than quietly building a second, dead drop-zone). Third BindRoot (RootImport) + Import list in BackupSpec, extended through ValidateBackupSpec/ClassifyBinds. Load-bearing: a stale `userdata: import/<app>` entry against the moved bind would be a WHOLE-BLOCK reject, taking the app's mandatory hdd classification with it. Exhaustive-root audit: resolveAbs/structuralGuard/ComputeCaptureSet/ ComputeFabBuckets now take importRoot explicitly (an import bind resolved against hddPath would name a directory on the wrong drive); unresolvable is refused loudly into Skipped. GetImportRoot added to both provider interfaces. Catalog-derived skeleton: UserdataSkeleton() -> UserdataSkeletonCarry() + BuildUserdataSkeleton(), SORTED. The carry-list makes zero-removals true by construction (`documents` is in no catalog app but on both boxes) and is the fresh-box floor. The sort is not tidiness: the naive map-order derivation measured 20 distinct outputs from 20 identical runs, which with fbNeedsRecreate is a fleet-wide FileBrowser restart loop. One authoritative compose parser: ParseComposeUserdataMounts now delegates to ParseComposeClassifiableBinds. Import root excluded from per-app migration. Surfaces: FileBrowser /srv/beolvasas source; app-page "Hova tegyem a fajlokat?" with PathEscape deep links (never QueryEscape) and class-driven copy; data_paths: annotation with the Fork-3 asymmetry; system-owned beolvasas SMB share refused server-side at handler AND store, button omitted in template. Caught on the way: the sharing template's row struct was function-local, so adding {{if .System}} would have 500'd every share row. ShareRow is now package-level and the render test uses the handler's own type. Tests 915 -> 949, all green. MinAgent unchanged.	2026-07-26 08:12:57 +02:00
admin	c97975c1df	v0.169.0: disk-health card + degradation notification (Lemezek állapota) ... Consumes the agent v0.94.0 smart payload (MinAgent floor unchanged; feature-detect by presence). One pure verdict fn agentapi.DiskVerdictFor shared by the dashboard card and the 6h check. Card via a 60s /disks TTL cache (anti-smartctl-storm); unreachable agent -> Nincs adat, page never blocks. disk-health-check (6h) emits disk_health_degraded on a degradation only vs an in-memory baseline (first run silent, recovery/UNKNOWN never notify, multi-attr -> one event). No global banner (deliberate). Pairs with the hub allowlist bump. Tests: verdict table (>=90 red-proof), notifier emit, check first-run-silent (red-proof), degradation-once, recovery-silent, UNKNOWN-excluded, FAILING-critical, nil-smart card, TTL cache.	2026-07-24 21:27:16 +02:00
admin	82c67e32e1	v0.168.0: customer-configurable backup window (Mentési időablak) ... ONE setting (window start W) drives every nightly leg at fixed, never-stored offsets: DB dump at W, tier-2 at W+60m, off-box at W+105m (wrap-safe). Precedence settings > controller.yaml db_dump_schedule > 02:30. - scheduler.UpdateDaily: retime a daily job at runtime (no restart) via a per-job buffered resched chan + a select case in runDailyJob. - new pure package internal/backupwindow (LegTimes/GateWindow/EffectiveWindow). - quiesce disk-tier window gate: scheduled cycles run only inside [W+2h,W+6h) with a safety valve (age>cadence+24h runs regardless); manual TriggerNow never gated. Backend.Due now also returns the backup age (from the agent's own /backup/due). - backup page: Mentési időablak card (time input + derived leg/gate rows); POST /backups/window validates -> saves -> UpdateDaily x3 -> flash. Tests: 5 groups, all red-proofed. Agent/cadence//backup/due untouched.	2026-07-24 20:55:44 +02:00