A service reports healthy because its process is alive, while WAL writes fail with ENOSPC. Log rotation also fails, retries amplify load, and deleting one file does not prove durable writes resumed.
Bounded local fault injection
mkdir -p ./disk-fixture
dd if=/dev/zero of=./disk-fixture/filler bs=1M count=900
# Run only inside a disposable volume with a declared size.
Drive a fixed write workload and record free bytes, successful writes, failed writes by error class, WAL/checkpoint age, retry count, and health state. Never run this against a workstation root or shared production volume.
| Threshold | Action |
|---|---|
| 20% free | warn and stop optional writes |
| 10% free | reject new work, retain read path |
| first ENOSPC | fail readiness, stop retries |
| space restored | integrity check before readiness |
A useful readiness probe performs a bounded durable-write check or observes a recent successful checkpoint; a process-only liveness probe should remain separate.
inject -> ENOSPC -> readiness false -> bounded cleanup
-> integrity/checkpoint -> test write -> readiness true
Test log rotation failure, deletion of an open file that does not release space, and a restart before checkpoint. Cleanup must remove the fixture, confirm file descriptors release blocks, run integrity checks, and prove a new write survives restart.
This local container drill does not model cloud-volume burst credits, remote filesystems, or every database WAL. Which signal should remove your instance from service first: free bytes, checkpoint age, or the first failed durable write?