What we built, what happens when things go wrong, and what is still missing. Generated from where-felhom-stands.yaml, which cites the capability map, register row or evidence document behind every claim — and which is checked by scripts/check_stands.py.
Seven stages, left to right, as they happen to a person.
1 Getting a box
A blank machine installs itself from our own boot image and registers itself as unclaimed — proven on two different boardsTwo boards: N100 2026-07-18, HP t740 2026-07-21.capability-map Bare-metal Felhom ISO (blank hardware → zero-touch auto-ins… · evidence tests/VALIDATION-n100-rehearsal-2026-07-18.md
The installer is published rather than pushed: rolling it back is one actchanged 2026-08-09, was walkedGate 6 of hostinstall_gates asserts the manifest names an installer-v tag; ran green tonight.why it moved: gate 6 asserts the manifest names an installer tag, but no walk of a rollback is on filecapability-map The installer is PUBLISHED, not pushed · register R-110
Installing onto hardware the customer already owns — the path exists, the first real one has not happenedUPGRADED: a real --mode byo install completed on demo-hp 2026-08-09 (Day-0 provision SUCCESS, 3m49s). Still not a customer's own hardware, so not 'walked' — but 'has not happened' is now false.evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
A machine that ends up on the wrong network port explains itself on screen and finds its way back — proven in a virtual drill, never on metalMap says the same: PROVEN-LIVE (nested drill — nested != metal).capability-map Box survives a wrong-NIC install · evidence audits/SPIKE-firstboot-nic-sweep-2026-07-22.md
Reinstalling a machine we previously installed refuses, because our own removal leaves a name service holding the port our own installer checksWarning card. Confirmed live 2026-08-09; dnsmasq restarted by --uninstall seizes :53.register R-272 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
2 Making it theirs
An emailed one-time code; the customer sets their own password and the operator never sees itExercised live 2026-08-09: reset code accepted at /claim, 302 + session.capability-map Customer claim: one-time emailed code → customer sets own p… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
The customer can bind their own machine from a link, with no operator present — done once, for realattempts=0, locked=0, source customer_selfbind, 2026-07-18.capability-map Customer binds their own appliance (self-service) · evidence tests/VALIDATION-n100-rehearsal-2026-07-18.md
It has never been done by a person who is not the operatorMap records MISSING (as evidence). Still true after 2026-08-09.capability-map A customer (not the operator) performs a restore via UI alone
The same word is used for two different secrets across three surfaces; the email points at a page a rebuilt machine does not showWarning card. Cost a real code on 2026-08-09.register R-282 · register R-283
3 Using it
Apps installed from a catalog of about 52, with a memory guard and health-aware progress53 templates listed on the rebuilt box 2026-08-09; deploy driven live.capability-map Deploy an app from the catalog (env config, memory guard, h… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
Start, stop, restart, update, logs, remove — and the parts that must not be stopped cannot bechanged 2026-08-09, was walkedwhy it moved: no walk document cited by the map row or anywhere elsecapability-map App lifecycle: start/stop/restart/update/logs/remove/redeploy · register R-108
Reachable from anywhere through a tunnel, per-app addressesVerified tonight: the rebuilt box answered on its public URL from outside.capability-map Remote access via Cloudflare Tunnel + Traefik (per-app subd… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
Reachable on the home network when the internet is downneeds-hardwareOnly an observation on a running box with WAN pulled could settle it. Boxes are off.capability-map LAN access when internet is down (lan_resolver)
Phone photos, documents with text recognition, files from Windows Explorer or a Maccapability-map Files from Windows Explorer / Mac Finder (SMB server) · evidence audits/SPIKE-lan-discovery-2026-07-18.md
A one-tap launcher, and a read-only guest link for visitorscapability-map Indítópult (app launcher) — one-tap grid of the household's…
Media to a TVMap: MISSING.capability-map Media to TV via DLNA
Separate accounts per household memberMap: MISSING.capability-map Multiple household users / per-person accounts
4 Drives
A new drive is found, offered, formatted, mounted and enrolled — including on awkward older boot layoutsmoved twice: 2026-08-09 walked -> built, then 2026-08-10 back to walkedApplies to a NEW drive. Re-attaching an existing one after a reinstall is R-280 and fails.why it moved: receipt found 2026-08-10: a live throwaway drive on felhom-pve walked scan -> format -> mount -> PVE dir storage -> one-click Regisztralas, with the resulting storage.cfg entry and systemd mount unit recorded. The map already read PROVEN-LIVE; the dataset was behind it. Scope unchanged: this is a NEW drive. Re-attaching an existing one after a reinstall is R-280 and still fails.capability-map Drive wizard: scan/format/mount/enroll, incl. legacy-boot L… · register R-220 · evidence audits/SPIKE-raw-drive-enroll-2026-06-15.md
Moving data between drives, crash-safe; removing a drive safely; unplug detectedchanged 2026-08-09, was walkedwhy it moved: no walk document citedcapability-map Data migration between drives (all / per-app), crash-safe
A network drive can be browsed and hold bulk media, but may not hold an app's data — enforcedRefuseAsAppNamespace is the fail-closed predicate.capability-map Network storage (NAS) is browse + bulk-media only · register R-108 · evidence audits/R108-network-app-namespace-2026-07-30.md
After a reinstall the data drive cannot be re-attached through any dashboard routeWarning card. /api/disks/candidates returns empty; the restore page promises two clicks.register R-280 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
5 Backing up
App data on the machine, nightly database dumps, a copy on a second drivemoved twice: 2026-08-09 walked -> built, then 2026-08-10 back to walkedTrue for every app but one, and that one was found on 2026-08-21: paperless-ngx's 72-table PostgreSQL was dumped nightly and written into a folder named after an app that does not exist, so it never entered the recovery unit, the second-drive copy or the off-site copy. Fixed and proven live 2026-08-22 (R-355) — the dump is now in the app's own unit and in the off-site snapshot for the first time. A catalogue-wide sweep, itself proven able to convict a planted case, says 1 of 53 was affected.why it moved: receipt found 2026-08-10: an adversarial, destructive, unattended overnight campaign across both boxes and ep0, with A2 (one quiesce, two tiers) proven end to end. Map already read PROVEN-LIVE.capability-map Tier-2 secondary-drive copy: class-driven legs · register R-355 · evidence audits/CAMPAIGN-8-backup-restore-2026-07-27.md · evidence audits/DRILL-backup-truth-2026-08-21/evidence/phase4-paperl…
A whole-machine archive that lands off the guest's own disk — a single-drive machine is recorded as degraded rather than pretendingchanged 2026-08-09, was walkedwhy it moved: no walk document citedcapability-map Whole-guest backup lands OFF the guest's own physical device · register R-165
An encrypted off-site copy, sealed with a key the operator cannot readchanged 2026-08-09, was walkedThe COPY is real and the key is still unreadable to us — what failed is DAILY and UNBROKEN. The 2026-08-09 note said '18 snapshots, daily, unbroken'; the next snapshot after 2026-08-09 08:30 was 2026-08-21 22:17, put there by hand during the drill. Twelve days, no alarm. Two causes in series: the 2026-08-21 rebuild lost the off-box target (R-193's shape), and after the self-heal restored it EVERY per-app switch was still off, so the first run logged 'backup OK: 0 app(s) backed up, 14s'.why it moved: the claim is about a CONTINUING daily copy; a 12-day silent gap was found on 2026-08-21 and nothing reported itcapability-map Offsite (restic → Hetzner Storage Box) · register R-199 · register R-193 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
The backups prove themselves: a restore is actually performed, unattended, on every tier, on both machineschanged 2026-08-09, was walkedSTILL GREY, and for a sharper reason than in August. The scheduler runs and the check works — it failed again on 2026-08-21, unprompted, and named the cause exactly: after the reinstall the box presents a different PBS key than its own older archives were sealed with, so those archives cannot be opened at all (R-366). A tier whose archives are orphaned is a different alarm from a tier whose test failed, and only the second is being said.proof decay: PROOF-DECAY RULE FIRED (first time it has). A receipt EXISTS - architecture/_recovery-inventory-2026-07-28.md carries live journal lines for scheduled restore-tests on both boxes and both tiers - but it is superseded by later observation: demo-hp logged restore_test_failed on 2026-08-05, and the box has since been wiped and reinstalled (2026-08-09). The claim is about a CONTINUING scheduled behaviour, so a 2026-07-28 observation cannot carry it. Stays grey until a scheduled restore-test is seen passing on the rebuilt box. THE CAPABILITY MAP STILL READS PROVEN-LIVE (2026-08-03) AND IS NOW THE THING OUT OF STEP.why it moved: no walk document cited, and the last recorded restore-test on demo-hp FAILED (2026-08-05)capability-map Restore-proof is UNATTENDED — the scheduler covers EVERY tier · register R-86 · register R-366 · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
The customer is warned before a drive fills, per drive, in their own languagechanged 2026-08-09, was walkedThe warning is real and was SEEN firing on 2026-08-21 with the right Hungarian copy, naming the drive and the free space. What 'BEFORE' cannot survive is the cadence: the watcher runs once a day at 03:30 plus once at startup (R-363), so a filesystem that fills at 03:31 goes unannounced for ~24 h. Watched live: the 69 GB volume carrying all 40-class app data was filled to 99% and the watcher said nothing, while the backup reserve was already refusing an app per run and telling the hub about it.why it moved: a daily check cannot carry the word BEFORE; observed silent for the whole window a filesystem sat at 99%capability-map The customer is warned BEFORE a filesystem fills · register R-167 · register R-363 · evidence audits/SPIKE-r165-mp1-merge-2026-08-02.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
A backup that covered nothing still calls itself successfulWarning card — and the drill found two more of it, both live. (1) An off-site run with no app selected logs 'backup OK: 0 app(s) backed up'; the card does say 'nincs kijelölt alkalmazás' beside the green tick, so this one is honest if you read past the tick. (2) A restore that placed nothing reported success: '0 fájl visszaállítva', ok=true. The second is the one that matters and it is R-353, still open. The volume half of it is fixed (R-354): the message now names what came back.register R-240 · register R-353 · register R-354 · evidence audits/DRILL-backup-truth-2026-08-21/evidence/phase3-experi…
6 When something goes wrong
The machine watches itself and repairs some faults without telling anyone it had tochanged 2026-08-09, was walkedR-264 records that the self-heal counters reach the hub and are decoded nowhere.why it moved: no walk document citedcapability-map Box survives an unattended app or guest-network failure · register R-264
Failures reach the operator by email, one mail per run, every failing app namedchanged 2026-08-09, was walkedCONFIRMED against source: backup_run_failures digest is allowlisted (api/handler.go:1837), operator-only (dispatcher.go:423), templated (templates.go:48); recovery_unit_capture_failed is record-only (dispatcher.go:376). NOTE: R-182's register row still describes the PRE-FIX behaviour — see R-289.why it moved: the digest is wired and source-verified, but no run of it has been observed deliveringcapability-map A failed per-app Tier-1 backup reaches the OPERATOR — EVERY… · register R-182
Failures reach the customerThe customer leg is built; today's log shows customer-channel rows skipped as operator_only.capability-map App crashes → customer notified (one event per transition, …
An already-paired box is still told to pair itselfWarning card.register R-214 · register R-235
7 Getting everything back
A rebuilt machine shows a full-page recovery screen without anyone looking for it, and says plainly that nobody can replace a lost recovery codeSeen unsought on the rebuilt demo-hp 2026-08-09, seal date matching host_escrow.created_at.register R-193 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
The customer's code opens the sealed package and the data returns byte for byte — including accented Hungarian filenames, verified as raw bytesStill true, and re-proven 2026-08-22 (5/5 byte-identical, both accented names as raw bytes) — but the SCOPE is narrower than the sentence sounds and was silently narrower still until v0.218.0. The drill found the off-site restore had NO named-volume leg at all: the archive sat in the unit, the snapshot and the checking folder and was never replayed, under a success message (R-354, fixed and proven 2026-08-22). AND 40 of the 53 catalogue apps STILL cannot run this route at all — it refuses first, saying a running app is not installed (R-356, open). So: proven for an app that declares a data drive; unproven and currently unreachable for the class whose entire dataset is a named volume.register R-201 · register R-354 · register R-356 · evidence tests/walk5-r201-2026-08-07/journal.md · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
Walked end to end with no command line inside the machine (2026-08-07)contestedCONTESTED-RESOLVED: not a contradiction. The walk proves the ROUTE needs no guest shell; the map's MISSING row is about a NON-OPERATOR doing it, which has still never happened. Two questions, one word 'customer'.register R-201 · evidence tests/walk5-r201-2026-08-07/journal.md · capability-map A customer (not the operator) performs a restore via UI alone
Putting restored files back where they belong is still manualWarning card. Confirmed 2026-08-09: the restore lands in a verification folder and says so.register R-213
The tripwire that says someone is opening this customer's backups does fireCONFIRMED tonight from the hub store: escrow_blob_served 2026-08-09 10:19:41Z = 12:19 CEST in the operator's mailbox. R-281's original claim of silence is WITHDRAWN.register R-281 · register R-285 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
19 situations, and what actually happens in each.
A disk starts failing
A disk starts failing — healthy path only; a genuinely failing disk has never been seenneeds-hardwareOnly a failing disk on a running box could settle it.capability-map Lemez-egészség felügyelet: per-disk SMART kártya · evidence audits/SPIKE-smart-coverage-2026-07-25.md
The backup drive is unplugged
The backup drive is unpluggedcapability-map An ABSENT backup-target drive raises its OWN alarm · evidence audits/R116-v0116-2026-07-30.md
A drive is filling up
A drive is filling upchanged 2026-08-09, was walkedwhy it moved: no walk document citedregister R-167
An app crashes
An app crashes — the email leg has never been confirmed end to endConsistent with fault.operator-email: the digest is wired but its delivery is unobserved.capability-map App crashes → customer notified (one event per transition, …
Power cut mid-backup
Power cut mid-backupevidence audits/AUDIT-power-outage-recovery-2026-07-22.md
The guest is destroyed
The guest is destroyedregister R-201 · evidence audits/DRILL-r201-night-run-2026-08-04.md · evidence audits/DRILL-r201-night-run-2026-08-04.md
The whole machine is wiped and reinstalled
The whole machine is wiped and reinstalled — the data comes backchanged 2026-08-09, was walkedThe 2026-08-09 rehearsal really did return 4/4 byte-identical, and that stands. What the next real reinstall showed (2026-08-21, demo-hp) is that the rebuild ORPHANS BOTH OFF-PREMISES TIERS at once, quietly: the restic target was lost and needed a self-heal plus a per-app re-enable before any copy resumed (R-193), and the PBS archives from before the reinstall cannot be opened by the rebuilt box at all, because it now presents a different key (R-366). The data came back in the rehearsal because the rehearsal restored it immediately; a machine left alone after a reinstall is not protected in the meantime and nothing says so.why it moved: a real reinstall on 2026-08-21 left both off-premises tiers broken - one silently for 12 days, the other unreadable - so 'the data comes back' holds only if someone restores it at onceregister R-193 · register R-366 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
The whole machine is wiped and reinstalled
The whole machine is wiped and reinstalled — the journey needs a terminal twiceCONTESTED-RESOLVED against the map: the map's PROVEN-LIVE row is scoped to a controller-data-volume REBUILD (2026-08-04), not a whole-host reinstall. The map has no row for the host case, so there was no contradiction — only a gap.register R-273 · register R-280 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
The machine is stolen, and someone opens the backups
The machine is stolen, and someone opens the backups — the operator is toldConfirmed 2026-08-09 from the store and the mailbox.register R-281 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
The customer forgets their dashboard password
The customer forgets their dashboard passwordExercised 2026-08-09 via the reset code.evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
The customer loses their recovery code
The customer loses their recovery code — by design, the data is unrecoverablemoved twice: 2026-08-09 walked -> built, then 2026-08-10 back to walkedThe recovery screen states it in Hungarian.why it moved: receipt found 2026-08-10, and then proven the hard way: the recovery screen states in Hungarian that nobody can replace a lost code, and on 2026-08-10 demo-felhom's pre-fix key was confirmed unrecoverable by construction (identity_blob NULL; the restic password lives only in that bundle). The by-design refusal is real.capability-map Escrow ceremony: customer-facing wizard, one-shot R claim, … · register R-198 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
The machine moves house / new network
The machine moves house / new networkcapability-map Box survives a site/network change (relocation, different s… · evidence audits/AUDIT-vacation-remote-ops-2026-07-20.md
The internet goes down
The internet goes down — built, never walkedneeds-hardwareNeeds a running box with WAN pulled.capability-map LAN access when internet is down (lan_resolver)
The hub is down
The hub is downchanged 2026-08-09, was walkedwhy it moved: no walk document citedcapability-map Config/state change round-trips in seconds (hub↔box immediacy
We ship a broken release
We ship a broken release — the guard is missingProven the hard way on 2026-08-09: a vouched agent version had no git tag and every install died at 5/8. Both guards still owed.register R-273 · register R-287
A fresh install picks up an old image
A fresh install picks up an old imageNarrowed 2026-08-09: the resume path fetched the vouched golden; a fresh install still takes the newest LOCAL archive with no manifest comparison.register R-274
The off-site provider account is deleted
The off-site provider account is deletedThe credential that holds the customer's documents can still delete.register R-95
The machine is switched off for an afternoon
The machine is switched off for an afternoon — no notion of expected downtimeConfirmed 2026-08-09: eight operator mails for deliberate, attended work.register R-285
A customer restores their own data with no help
A customer restores their own data with no helpcontestedCONTESTED-RESOLVED: the map's MISSING is about a NON-OPERATOR performing it; the walks prove the route, not the person. Neither record was wrong. SINCE 2026-08-21 there is a second, harder blocker and it is not about the person at all: for the 40 of 53 apps that declare no data drive the off-site restore REFUSES before it starts, telling the customer a running app 'nincs telepítve' and to reinstall it to the same place — which those apps give them no way to choose (R-356). Those are exactly the apps whose whole dataset is a named volume. Until that is fixed, most customers cannot self-restore off-site even in principle.capability-map A customer (not the operator) performs a restore via UI alone · register R-201 · register R-356 · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md