Card 07/ 08
All 8 cards
ExerciseDifficulty: Advanced1 min
Four Situations: Name the Failure in Each
Four descriptions from four incident reports. Name the failure in each and say what would fix it. All four before the reveal.
| What was observed | |
|---|---|
| 1 | Eight request threads BLOCKED, processor at two per cent, no progress for an hour |
| 2 | Processor at ninety per cent, two threads logging "retrying" thousands of times a second |
| 3 | Nine of ten workers finishing normally; one has not completed a task since start-up |
| 4 | Threads recover after five seconds, transfers fail under load, no hang |
The four failures, and what fixes each
| Failure | Fix | |
|---|---|---|
| 1 | Deadlock | A global lock ordering, applied everywhere both locks are taken |
| 2 | Livelock | Randomised backoff, or one thread that decides rather than both yielding |
| 3 | Starvation | A fair lock, or work-stealing so no thread depends on winning a race |
| 4 | Deadlock, with a timeout on top | Order the locks; keep the timeout as the net |
Processor use separates the first two, which is why it is the first thing to look at. Idle and stuck is a cycle of locks; busy and stuck is threads reacting to each other.
The fourth is the one that gets closed as a different bug. Intermittent failures under load with no hang looks like a capacity problem, and it is the same cycle with a net under it.