docs(R-117): the hang case measured — an I/O probe turns a wedged drive into an unkillable agent
Completes the spike once the venue came back. Q4's hang case and teardown are now measurements, not plans. Against a dmsetup-suspended device (I/O queues instead of returning EIO): - P1 (devno compare) and P2 (ext4 abort flags) completed in 364us / 206us. They read /proc, so no block device is involved. - statfs and getdents completed and reported HEALTHY — on a wedged device they do not even hang. R-117b confirmed in a second failure mode. - EVERY probe that touches the device blocked, including a buffered write with no fsync: the O_CREAT metadata path needs journal access (wchan=do_get_write_access). There is no cheap-and-safe write probe. - The blocked process survived SIGTERM AND SIGKILL (stat=D, wchan=folio_wait_bit_common, still alive 3m50s after kill -9) and died only when the device was resumed. So `systemctl restart felhom-agent` would hang, leaving the agent unrecoverable until the device returns or the host reboots. The thread count does not reveal the leak (5->5, 5->6). Filed as R-117f. A timeout protects the caller's control flow and nothing else, so "the fix must issue no block I/O" is now a fence rather than a preference — the thread-leak hypothesis the probes were built to test turned out to be the weaker half of the result. Teardown done, all three layers: guest 9301 destroyed, r117scratch removed, both dm and both loop devices gone, scsi_debug unloaded, local back to 37.02% against a 37.00% session start. Fences re-verified AFTER teardown: 9201 running, drill-r50 stopped, local-lvm 38.84% byte-identical, felhom-backup content unchanged, live /mnt/felhom-drives intact with both submounts, agent active. Layer 3 genuinely empty — 9301 had no NIC and ran no controller. Trap recorded: a suspended dm device must be resumed BEFORE any umount, or the teardown blocks on the same uninterruptible sleep.
This commit is contained in:
+39
-17
@@ -27,7 +27,7 @@ the source of a behavioural claim — every predicate result came from a probe b
|
||||
|
||||
## Results
|
||||
|
||||
All seven questions answered empirically except the hang case (below).
|
||||
**All seven questions answered empirically.**
|
||||
|
||||
- **Q1 — reproduced 3/3**, two device classes, on a purpose-built scratch LXC (9301) on demo-hp.
|
||||
**The device-node change is a consequence of the defect, not a precondition** — control test: with
|
||||
@@ -39,8 +39,9 @@ All seven questions answered empirically except the hang case (below).
|
||||
- **Q3 — confirmed at source and live.** Both halves of the R-113 conjunction compare **field 5** of a
|
||||
mountinfo line and **never read field 3 (`major:minor`)**, so neither can see that the bind and the
|
||||
raw mount name different devices. Measured `BoundUnderParent = TRUE` over an `EIO` namespace.
|
||||
- **Q4 — a pure-`/proc` check costs 0.17–0.45 ms**, cannot hang, spins up no disk, writes nothing.
|
||||
**`statfs` and `getdents` both SUCCEED on a dead namespace** — probes built on either are hollow.
|
||||
- **Q4 — a pure-`/proc` check costs 0.16–0.45 ms**, cannot hang, spins up no disk, writes nothing.
|
||||
**`statfs` and `getdents` both SUCCEED on a dead namespace** — probes built on either are hollow. The
|
||||
hang case is below; it is the sharpest result in the run.
|
||||
- **Q5 — the agent**, and not on balance: the controller runs inside the guest and cannot see the host
|
||||
mount tables the check needs.
|
||||
- **Q6 — recovery works in place, guest never restarted** (init PID identical). **The repair code
|
||||
@@ -49,26 +50,47 @@ All seven questions answered empirically except the hang case (below).
|
||||
- **Q7 — a bind can die in steady state, no cycle at all.** The gate produces no action and **nothing
|
||||
is emitted on any channel.** A `Return`-branch fix cannot reach this half.
|
||||
|
||||
## Not established
|
||||
## Q4's hang case — measured, and it is the sharpest result
|
||||
|
||||
**The hang case (findings §6.3).** The venue (`dmsetup suspend --noflush`) and the two blocking probes
|
||||
(write+`fsync`, cache-dropped read, both with timeout-fires and thread-leak reporting) were built; the
|
||||
run did not happen — the venue lost internet mid-transfer of the rebuilt binary. **The associated
|
||||
thread-leak hypothesis is explicitly not claimed as a result.** It does not gate the `/proc`
|
||||
recommendation but does gate adding any I/O confirmation probe.
|
||||
Against a `dmsetup suspend`ed device (I/O queues instead of returning `EIO`):
|
||||
|
||||
## Teardown — OWED
|
||||
- **P1 and P2 completed in 364 µs / 206 µs.** They read `/proc`, so no block device is involved.
|
||||
- **`statfs` and `getdents` completed and reported HEALTHY** — on a wedged device they do not even hang.
|
||||
- **Every probe that touches the device blocked, including a buffered write with no `fsync`** — the
|
||||
`O_CREAT` metadata path needs journal access (`wchan=do_get_write_access`). There is no cheap-and-safe
|
||||
write probe.
|
||||
- **The blocked process survived `SIGTERM` and `SIGKILL`** (`stat=D`, still alive 3m50s after `kill -9`)
|
||||
and died only when the device was resumed. So **`systemctl restart felhom-agent` would hang**, leaving
|
||||
the agent unrecoverable until the device returns or the host reboots. The thread count does not reveal
|
||||
the leak (5→5, 5→6).
|
||||
|
||||
**Not performed** — the second site outage began first. Findings §14 lists the exact commands. Nothing
|
||||
fenced was touched at any point (9201 `running`, `drill-r50` `stopped`, `local-lvm` untouched,
|
||||
`felhom-backup` not re-targeted, `/mnt/felhom-drives` untouched, ep0 read-only). **Layer 3 is
|
||||
genuinely empty** — guest 9301 had no network interface and ran no controller, so no hub-side record
|
||||
was created.
|
||||
**A timeout protects the caller's control flow and nothing else.** This turns "prefer a cheap probe" into
|
||||
a fence: **the fix must issue no block I/O.**
|
||||
|
||||
## Teardown — done, all three layers
|
||||
|
||||
Guest 9301 destroyed; `r117scratch` removed; both dm devices and both loop devices gone; `scsi_debug`
|
||||
unloaded (`/dev/sd*` back to `sda1..3`); no `r117` mounts, `/mnt` and `/root` exactly as found; `local`
|
||||
**37.02 %** against a session-start **37.00 %**. Fences re-verified *after* teardown: 9201 `running`,
|
||||
`drill-r50` `stopped`, `local-lvm` **38.84 % byte-identical**, `felhom-backup` `content backup`
|
||||
unchanged, live `/mnt/felhom-drives` intact with both submounts, agent service `active`. **Layer 3 is
|
||||
genuinely empty** — 9301 had no network interface and ran no controller, so no hub-side record was ever
|
||||
created.
|
||||
|
||||
**Ordering trap worth keeping:** a suspended dm device must be `dmsetup resume`d *before* any `umount`,
|
||||
or the teardown itself blocks on the same uninterruptible sleep.
|
||||
|
||||
## Not measured
|
||||
|
||||
No load or duration testing of the recommended check — P1/P2 were single calls, not a sustained
|
||||
reconcile loop on a many-drive box. Nothing suggests a problem (they are two `/proc` reads the code
|
||||
already performs), but it was not measured.
|
||||
|
||||
## Findings filed, none fixed
|
||||
|
||||
R-117 (mechanism + recipe), **R-117a** steady-state death with no event (HIGH, larger than R-117 as
|
||||
filed), **R-117b** `statfs`/`getdents` are hollow liveness probes, **R-117c** three untested comments
|
||||
promising "live + usable in the guest", **R-117d** the self-heal that already exists is short-circuited
|
||||
(HIGH), **R-117e** both demo boxes share one failure domain — no route survives the site losing
|
||||
internet, including the WireGuard OOB path.
|
||||
(HIGH), **R-117e** both demo boxes share one failure domain — no route survives the site losing internet,
|
||||
including the WireGuard OOB path, **R-117f** an I/O liveness probe turns a wedged drive into an
|
||||
unkillable agent (HIGH — disqualifies a whole probe class).
|
||||
|
||||
@@ -12,7 +12,7 @@ Output is this document. Findings are filed as `R-n`, none fixed.
|
||||
| Q1 | Reproduce deliberately | **YES — 3/3 attempts, two independent device classes.** A device-node change is **NOT a precondition — it is a CONSEQUENCE** of the stale bind, proven by a control test |
|
||||
| Q2 | What is dead, at which layer | The **guest bind and the host bind** (same mount, one peer group); the **raw host mount is healthy on the new device**. **Two distinct death states** with different errnos — `EIO`(5) both directions, and `EROFS`(30) on write only |
|
||||
| Q3 | Why `boundUnderParent` returns true | It is a **string comparison of field 5 of a mountinfo line**. It never touches the filesystem, and **neither half reads field 3 (the device number)** — so neither can see that the two entries name different devices |
|
||||
| Q4 | Cost + failure modes of a real check | A **pure-`/proc` check costs 0.19–0.45 ms, cannot hang, cannot spin up a disk and writes nothing.** The two cheapest *I/O* probes — `statfs` and `getdents` — **succeed over a fully dead filesystem** and are worthless. The hang case was **NOT measured** (§6.3) |
|
||||
| Q4 | Cost + failure modes of a real check | A **pure-`/proc` check costs 0.16–0.45 ms, cannot hang, cannot spin up a disk and writes nothing.** The two cheapest *I/O* probes — `statfs` and `getdents` — **succeed over a fully dead filesystem** and are worthless. **Every probe that reaches the device blocks on a wedged one, and the process then cannot be killed even with `SIGKILL`** (§6.3) |
|
||||
| Q5 | Where the check belongs | **The agent**, and not on balance — the controller **cannot** perform it: the inputs are host mount tables it has no access to |
|
||||
| Q6 | In-place recovery possible | **YES — proven, guest never restarted** (init PID identical across the repair). And **the repair code already exists, is already invoked at exactly the right moment, and is short-circuited by the same false predicate** |
|
||||
| Q7 | Scope — steady state too? | **YES, and this is the worse half.** A drive can die **with no detach/return cycle at all**; the gate then never runs, the `Return` branch never fires, and **no event of any kind is emitted** |
|
||||
@@ -358,9 +358,9 @@ All timings from the R-117 state on drive B, on the venue, one run each, `-timeo
|
||||
| **P2 ext4 abort options** | `shutdown` **or** `emergency_ro` in the super options — **pure `/proc`** | **DEAD=true** ✅ | **DEAD=true** ✅ | false ✅ | **0.17–0.24 ms** | **Catches both.** ext4-specific |
|
||||
| P3 `statfs` | one syscall | **OK — MISSES IT** ❌ | OK — misses ❌ | OK | 0.042 ms | **Worthless** |
|
||||
| P4 `getdents` | read one dir entry | **OK — MISSES IT** ❌ | OK — misses ❌ | OK | 0.045–0.061 ms | **Worthless** |
|
||||
| P5 write + unlink | create, write 1 byte, remove | `EIO`(5) ✅ | `EROFS`(30) ✅ | OK | 0.042–0.152 ms | Catches both, **but writes**, and cannot distinguish a legitimately-`ro` mount from a dead one |
|
||||
| P6 write + **fsync** | forces the byte to the device | **not measured** (§6.3) | — | — | — | **The probe that can hang** |
|
||||
| P7 read with page cache dropped (`POSIX_FADV_DONTNEED`) | forces a read to reach the device | **not measured** (§6.3) | — | — | — | **The probe that can hang** |
|
||||
| P5 write + unlink | create, write 1 byte, remove | `EIO`(5) ✅ | `EROFS`(30) ✅ | OK | 0.042–0.152 ms | Catches both, **but writes**, cannot distinguish a legitimately-`ro` mount from a dead one, **and BLOCKS on a wedged device** (§6.3) |
|
||||
| P6 write + **fsync** | forces the byte to the device | — | — | OK, **11.4 ms** | **11.4 ms healthy** | **BLOCKS on a wedged device.** 25–70× the `/proc` probes even when healthy, on a loop device with no seek cost |
|
||||
| P7 read with page cache dropped (`POSIX_FADV_DONTNEED`) | forces a read to reach the device | — | — | OK, 0.18 ms | 0.18 ms | **BLOCKS on a wedged device** |
|
||||
|
||||
`GuestSeesMount` itself costs **3.85–5.03 ms**, an order of magnitude more than any probe above,
|
||||
because it forks `lxc-info`. **Any of P1/P2 is free relative to the check it would strengthen.**
|
||||
@@ -380,26 +380,75 @@ because it forks `lxc-info`. **Any of P1/P2 is free relative to the check it wou
|
||||
This is the strongest argument for the pure-`/proc` pair over anything else, independent of the hang
|
||||
question.
|
||||
|
||||
### 6.3 The hang case — NOT ESTABLISHED
|
||||
### 6.3 The hang case — MEASURED, and it is worse than "a probe that hangs"
|
||||
|
||||
**This is the one question in the brief that this run does not answer, and the gap is real.**
|
||||
Venue: the `dm-linear` drive suspended with `dmsetup suspend --noflush --nolockfs`, which **queues all
|
||||
I/O indefinitely instead of returning `EIO`** — the controlled "half-dead or spun-down device" the brief
|
||||
asks about. Verified suspended (`dmsetup info -o attr` → `L-sw`, the `s`), then the full probe set with
|
||||
`-timeout 3s`, then `dmsetup resume`.
|
||||
|
||||
Designed and built, not run: a `dm-linear` device suspended with `dmsetup suspend --noflush`, which
|
||||
queues all I/O indefinitely instead of returning `EIO` — the controlled "half-dead or spun-down device"
|
||||
the brief asks about. `r117probe` was extended with **P6 (write + `fsync`)** and **P7 (read with the page
|
||||
cache dropped)** — the two probes that must reach the platter and therefore can block — plus reporting of
|
||||
whether the blocked goroutine **ever returns** and the process's OS-thread count before and after.
|
||||
```
|
||||
=== PROBES against a WEDGED (not erroring) device, timeout 3s ===
|
||||
HALF1 GuestSeesMount = true [3.983519ms]
|
||||
HALF2 isHostMountpoint = true [234.181µs]
|
||||
=====> BoundUnderParent = true
|
||||
P1 devno-compare raw="252:12" bind="252:12" -> STALE=false [363.878µs] <-- COMPLETED
|
||||
P2 ext4-abort-opts superopts="rw" -> DEAD=false [206.104µs] <-- COMPLETED
|
||||
P3 statfs COMPLETED in 40µs err=<nil> <-- reports HEALTHY
|
||||
P4 getdents COMPLETED in 64µs err=<nil> <-- reports HEALTHY
|
||||
P6 write+fsync TIMED OUT after 3.002s — caller proceeds; goroutine still blocked
|
||||
P6 ...still blocked 8s in; threads 5 -> 5 (LEAKED — the syscall is uninterruptible)
|
||||
P7 read-uncached TIMED OUT after 3.002s — caller proceeds; goroutine still blocked
|
||||
P7 ...still blocked 8s in; threads 5 -> 5 (LEAKED)
|
||||
P5 write+unlink TIMED OUT after 3.002s — caller proceeds; goroutine still blocked
|
||||
P5 ...still blocked 8s in; threads 5 -> 6 (LEAKED)
|
||||
```
|
||||
|
||||
**It was not run: the second house-internet outage (§2.5) began while the rebuilt 3.6 MB binary was being
|
||||
copied to the venue.** The build succeeded locally; the transfer did not complete.
|
||||
**Four results, in ascending order of severity.**
|
||||
|
||||
Not measured, therefore not claimed. What is *known* from what did run:
|
||||
1. **P1 and P2 completed normally, in microseconds.** They read `/proc/self/mountinfo`, which involves
|
||||
no block device. **This is the whole argument for the §11 recommendation**: on a wedged device the
|
||||
pure-`/proc` check is not merely faster, it is the only one that returns at all.
|
||||
2. **P3 and P4 completed *and reported healthy*.** On a wedged device they do not even have the decency
|
||||
to hang — they lie, exactly as they do on a dead one (§6.1). Confirms **R-117b** in a second failure
|
||||
mode.
|
||||
3. **Every probe that touches the device blocked — including P5.** P5 is a buffered `write` with no
|
||||
`fsync`, which was *expected* to complete out of page cache. It did not: the `O_CREAT` metadata path
|
||||
needs journal access (`wchan=do_get_write_access`). **There is no such thing as a cheap write probe
|
||||
that is safe on a wedged device.**
|
||||
4. **The process could not be killed — not by `SIGTERM`, not by `SIGKILL`.** This is the finding that
|
||||
changes the recommendation from a preference into a fence:
|
||||
|
||||
- **P1 and P2 cannot hang** — they read `/proc/self/mountinfo`, which involves no block device. This is
|
||||
a property of the files they read, and it is why the recommendation in §8 rests on them.
|
||||
- The reasoning that a timeout around a blocked uninterruptible syscall leaves a **leaked thread per
|
||||
poll** — and that at the reconcile ticker's 20-second cadence (`main.go:973`) that is 180 per hour —
|
||||
is the hypothesis P6/P7 were built to test. **It is untested. It must not be quoted as a result.**
|
||||
```
|
||||
# after `timeout 200` had already sent SIGTERM:
|
||||
PID STAT WCHAN ELAPSED COMMAND
|
||||
2493045 Dl folio_wait_bit_common 03:20 r117probe
|
||||
tid 2493045 stat=D wchan=folio_wait_bit_common
|
||||
tid 2493049 stat=D wchan=do_get_write_access
|
||||
|
||||
# explicit SIGKILL:
|
||||
kill -9 2493045
|
||||
after SIGKILL: 2493045 Dl folio_wait_bit_common 03:50 <-- STILL ALIVE
|
||||
|
||||
# only the device recovering released it:
|
||||
dmsetup resume r117cel2 -> probe still alive? no — it died once I/O completed
|
||||
```
|
||||
|
||||
**A timeout protects the caller's control flow and nothing else.** The goroutine and its OS thread are
|
||||
consumed permanently, and the *process* enters uninterruptible sleep. Applied to the agent, whose
|
||||
reconcile ticker runs every 20 s (`main.go:973`):
|
||||
|
||||
- the report loop's caller proceeds, so the timeout "works" — and the evidence of the problem is a
|
||||
thread you cannot see and cannot reclaim;
|
||||
- **`systemctl restart felhom-agent` would hang**, because `SIGKILL` does not reach a `D`-state thread.
|
||||
The agent becomes unrecoverable **until the device recovers or the host reboots** — on a box whose
|
||||
whole purpose is to be recovered remotely;
|
||||
- the thread count is **not** a reliable tell (5→5, 5→5, 5→6): the Go runtime reuses threads, so the
|
||||
leak does not announce itself in the one metric an operator might watch.
|
||||
|
||||
> **This disqualifies I/O probing on this path outright.** Not "use a timeout", not "use it carefully" —
|
||||
> a liveness probe that issues block I/O converts a dead drive into an unkillable agent. §11's
|
||||
> recommendation issues none.
|
||||
|
||||
---
|
||||
|
||||
@@ -623,9 +672,12 @@ no write, and no possibility of blocking the report loop.** Effectively free.
|
||||
promising a property nothing currently tests; the fix owes one that pins it.
|
||||
6. **Q7 needs its own scenario.** A test that only exercises detach→return will pass while the silent
|
||||
half stays broken. The in-place recipe in §10 is the second scenario.
|
||||
7. **The hang question is open** (§6.3). It does not gate this recommendation — P1 and P2 read `/proc`
|
||||
and cannot block — but it **does** gate any proposal to add an I/O confirmation probe on top. Settle
|
||||
§6.3 before accepting one.
|
||||
7. **Do not add an I/O confirmation probe. Measured, not preferred** (§6.3). On a wedged device every
|
||||
probe that touches it blocks — including a buffered write with no `fsync` — and the process then
|
||||
**cannot be killed by `SIGKILL`**, so the agent could not be restarted until the device recovered or
|
||||
the host rebooted. A timeout protects the caller's control flow and nothing else. **P1 and P2 both
|
||||
returned in microseconds on that same wedged device**; the recommendation issues no block I/O, and
|
||||
that is now a fence rather than a preference.
|
||||
|
||||
---
|
||||
|
||||
@@ -662,7 +714,8 @@ no write, and no possibility of blocking the report loop.** Effectively free.
|
||||
|---|---|---|---|
|
||||
| **R-117** | *(existing row, now with a mechanism and a recipe)* The bind-liveness signal is a path-presence test; the R-117 state is `BoundUnderParent = true` over an `EIO` namespace | **HIGH** | §3, §5 |
|
||||
| **R-117a** | **A drive's bind can die in STEADY STATE, with no detach/return cycle, and no event is emitted on any channel.** The gate produces neither action, so a `Return`-branch fix cannot reach it. **P1 does not detect it; only the filesystem's abort flag does** | **HIGH — larger than R-117 as filed** | §9 |
|
||||
| **R-117b** | **`statfs` and `getdents` both SUCCEED on a namespace returning `EIO` for all content.** Any liveness probe built on either is hollow by construction — it reports healthy in exactly the state it exists to detect | **MEDIUM (design trap)** | §4, §6.1 |
|
||||
| **R-117b** | **`statfs` and `getdents` both SUCCEED on a namespace returning `EIO` for all content** — and also on a **wedged** one, where they do not even hang. Any liveness probe built on either is hollow by construction — it reports healthy in exactly the state it exists to detect | **MEDIUM (design trap)** | §4, §6.1, §6.3 |
|
||||
| **R-117f** | **A liveness probe that issues block I/O turns a wedged drive into an UNKILLABLE AGENT.** On a suspended device every I/O probe blocked — including a buffered write with no `fsync` — and the process survived `SIGTERM` **and `SIGKILL`**, dying only when the device recovered. `systemctl restart felhom-agent` would therefore hang, leaving the agent unrecoverable until the device returns or the host reboots. **A timeout protects the caller's control flow and nothing else**, and the thread count does not reveal the leak (5→5, 5→6) | **HIGH (constraint on the fix — it disqualifies an entire probe class)** | §6.3 |
|
||||
| **R-117c** | **Three comments** (`disks.go:83`, `:154-155`, `:155-165`) state that `BoundUnderParent` means "live + usable in the guest". **Nothing tests it, and it is false in the measured state.** Not counted as a shipped false invariant because `:158-165` scopes its own claim narrowly — but the summary lines do not | **MEDIUM** | §5.3 |
|
||||
| **R-117d** | **`AttachDrive` logs and returns `"fully live"` for a namespace that is `EIO` on every call** (`intermediary.go:235`), disabling all three wired repair paths including the controller's pre-restart re-attach | **HIGH — this is the self-heal that already exists** | §8.2 |
|
||||
| **R-117e** | **The two demo boxes share one uplink and one failure domain.** Both went unreachable together, twice in one session, on all four known routes including the WireGuard OOB path via ep0. There is no route to either box that survives the site losing internet — the "out-of-band" path is not out-of-band with respect to this failure | **LOW (operational, not a code defect)** | §2.5 |
|
||||
@@ -674,50 +727,57 @@ about.** Whoever fixes R-117 will need that box on a current agent to validate l
|
||||
|
||||
---
|
||||
|
||||
## 14. Teardown — **OWED, NOT YET DONE**
|
||||
## 14. Teardown — DONE, all three layers
|
||||
|
||||
**The second internet outage (§2.5) began before teardown. This section is a liability, not a record.**
|
||||
Everything below is inert scratch on a Tier 0 box; nothing fenced was touched at any point (§14.2).
|
||||
Run after the venue came back; the site outage delayed it, it did not skip it.
|
||||
**Ordering note, recorded because it is a real trap:** any suspended dm device must be `dmsetup resume`d
|
||||
**before** any `umount`, or every unmount in the teardown blocks on the wedged device — the same
|
||||
uninterruptible-sleep mechanism as §6.3. Both devices were verified `attr=L--w` (not suspended) first.
|
||||
|
||||
### 14.1 Outstanding on demo-hp — the exact commands
|
||||
### 14.1 Layer 1 — the machine
|
||||
|
||||
```bash
|
||||
# layer 1 — the machine
|
||||
pct stop 9301 && pct destroy 9301 --purge
|
||||
# layer 2 — the host
|
||||
umount /mnt/r117-drives/sd /mnt/r117-drives/cel 2>/dev/null
|
||||
systemctl stop mnt-r117sd.mount mnt-r117cel.mount
|
||||
umount /mnt/r117-drives # the scratch shared parent self-bind
|
||||
dmsetup remove r117cel r117cel2 # now unheld once the binds are gone
|
||||
losetup -d /dev/loop0
|
||||
modprobe -r scsi_debug # removes /dev/sdb (the fake disk) entirely
|
||||
pvesm remove r117scratch && rm -rf /var/lib/vz/r117-scratch
|
||||
rmdir /mnt/r117-drives/sd /mnt/r117-drives/cel /mnt/r117-drives /mnt/r117cel /mnt/r117sd
|
||||
rm -rf /root/r117 /root/setup1.sh /root/setup2.sh /root/setup3.sh \
|
||||
/root/capture.sh /root/capture2.sh /root/r117probe /root/r117-pveam.log
|
||||
# optional: rm /var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst (124 MB, downloaded
|
||||
# this run; harmless to keep and useful for the next scratch guest)
|
||||
# then verify: pct list | pvesm status | dmsetup ls | losetup -a | lsmod | grep scsi_debug | findmnt /mnt/r117-drives
|
||||
```
|
||||
BEFORE pct list -> 9201 running demo-hp / 9301 running r117
|
||||
pct stop 9301 ; pct destroy 9301 --purge ("purging CT 9301 from related configurations..")
|
||||
AFTER pct list -> 9201 running demo-hp <-- only the pre-existing guest
|
||||
```
|
||||
|
||||
### 14.2 Layer 3 — the hub: **nothing to dispose of**
|
||||
### 14.2 Layer 2 — the host, with the space actually returned
|
||||
|
||||
| | Before | After |
|
||||
|---|---|---|
|
||||
| `pvesm status` | `r117scratch` **active**, `local` **38.97 %** | `r117scratch` **removed**, `local` **37.02 %** *(session start: 37.00 %)* |
|
||||
| `df -h /var/lib/vz` | 16G used, **42 %** | 15G used, **40 %** *(session start: 40 %)* |
|
||||
| `dmsetup ls` | `r117cel (252:11)`, `r117cel2 (252:12)` + 11 pre-existing | **both gone**; the 11 pre-existing `pve-*` unchanged |
|
||||
| `losetup -a` | `/dev/loop0` (cel.img), `/dev/loop1` (9301 rootfs) | **empty — no loop devices** |
|
||||
| `scsi_debug` | loaded (1) | **unloaded (0)**; `/dev/sd*` back to `sda1..3` only |
|
||||
| r117 mounts | raw + bind + scratch shared parent | `findmnt \| grep r117` → **(none)** |
|
||||
| `/mnt` | `+ r117-drives`, `r117cel`, `r117sd` | **`felhom-drives`, `nvme-1tb`, `vzsnap0`** — exactly as found |
|
||||
| `/root` | 7 scratch files + `r117/` | `ls /root \| grep -i r117` → **(none)** |
|
||||
|
||||
The transient `mnt-r117cel.mount` / `mnt-r117sd.mount` units died with the unmounts
|
||||
(`Unit mnt-r117sd.mount not loaded` — created by `systemd-mount`, so nothing was left in `/etc`). The
|
||||
124 MB `debian-13-standard` template was also removed.
|
||||
|
||||
### 14.3 Layer 3 — the hub: nothing to dispose of
|
||||
|
||||
**This run created no hub-side record.** No customer, no appliance, no pairing, no claim: guest 9301 ran
|
||||
no controller and had **no network interface at all**, and no agent was enrolled against it. Stated
|
||||
explicitly per §13's rule that "teardown clean" without layer 3 is the `sess-c` failure — here layer 3 is
|
||||
genuinely empty, and that is a different claim from silence.
|
||||
explicitly per §13 of the template — "teardown clean" without layer 3 is the `sess-c` failure. Here layer
|
||||
3 is genuinely empty, which is a different claim from silence.
|
||||
|
||||
### 14.3 Fences — verified untouched throughout
|
||||
### 14.4 Fences — re-verified after teardown, not assumed
|
||||
|
||||
| Fence | State |
|
||||
|---|---|
|
||||
| demo-hp guest **9201** | `running` throughout; never a probe target |
|
||||
| demo-hp **`drill-r50`** (VM 300) | `stopped`; never started, never touched |
|
||||
| demo-hp **`local-lvm`** | untouched — scratch rootfs deliberately placed on a new dir storage on `pve-root` |
|
||||
| demo-hp **`felhom-backup`** / `/mnt/nvme-1tb` | not re-targeted, not written to, `content backup` unchanged |
|
||||
| demo-hp guest **9201** | **`running`** — before, during and after; never a probe target |
|
||||
| demo-hp **`drill-r50`** (VM 300) | **`stopped`** — never started, never touched |
|
||||
| demo-hp **`local-lvm`** | **38.84 %** — byte-identical before and after, and to the session-start reading |
|
||||
| demo-hp **`felhom-backup`** | **`active`, 0.46 %**, `content backup` **unchanged** — not re-targeted, not written to |
|
||||
| live shared parent `/mnt/felhom-drives` | intact with **both** submounts (`nvme-1tb` on `/dev/nvme0n1[/felhom-data]`, `Felhom-Share` autofs) — a **separate** scratch parent was used throughout |
|
||||
| `felhom-agent` service on demo-hp | **`active`** |
|
||||
| demo-felhom, Peti's cluster | never contacted |
|
||||
| **ep0** | **read-only** — one `ssh`, `wg show`, and TCP reachability probes toward `10.77.0.2/.3` during outage 1. Nothing written |
|
||||
| `/mnt/felhom-drives` (live shared parent) | untouched; a **separate** scratch parent was used |
|
||||
| **Code** | **no `.go` file in either repo modified**; nothing built for deployment, nothing published, no version bumped |
|
||||
| Repos | the probe lives outside both, in the session scratchpad; the agent tree copy verified byte-identical to `d4eb259` |
|
||||
|
||||
@@ -725,12 +785,14 @@ genuinely empty, and that is a different claim from silence.
|
||||
|
||||
## 15. What did not happen, and why
|
||||
|
||||
- **Q4's hang measurement was not taken** (§6.3) — the venue went offline mid-transfer. The venue
|
||||
(`dmsetup suspend --noflush`) and the two probes (P6 write+`fsync`, P7 cache-dropped read, both with
|
||||
timeout-fires reporting and thread-leak counting) are built and specified; only the run is missing.
|
||||
**The associated hypothesis — that a timeout around a blocked syscall leaks a thread per poll — is
|
||||
explicitly NOT claimed as a result.**
|
||||
- **Teardown was not performed** (§14). Commands recorded; nothing fenced was touched.
|
||||
- **Everything the brief asked for was measured.** The two items the first internet outage deferred —
|
||||
Q4's hang case and teardown — were completed once the site came back: §6.3 and §14 are measurements,
|
||||
not plans. The thread-leak hypothesis P6/P7 were built to test turned out to be **the weaker half of
|
||||
the real result**: the leak is invisible in the thread count (5→5, 5→6), while the process being
|
||||
unkillable by `SIGKILL` is unambiguous.
|
||||
- **No load or duration testing of the recommended check.** P1/P2 were measured as single calls
|
||||
(0.16–0.45 ms), not across a sustained reconcile loop on a box with many drives. Nothing suggests a
|
||||
problem — they are two `/proc` reads the code already performs — but it was not measured.
|
||||
- **No fresh `/disks` payload and no fresh controller gate log** — deliberate, and the same call
|
||||
`DIAG-r116` §14 made: both were already captured verbatim on real hardware (twice and four times
|
||||
respectively), and running a controller here would have meant a hub customer, a pairing, a golden fetch
|
||||
|
||||
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user