Card 07/ 08

ExerciseDifficulty: Advanced1 min

Four Situations: Name the Failure in Each

Four descriptions from four incident reports. Name the failure in each and say what would fix it. All four before the reveal.

The four incidents
What was observed
1Eight request threads BLOCKED, processor at two per cent, no progress for an hour
2Processor at ninety per cent, two threads logging "retrying" thousands of times a second
3Nine of ten workers finishing normally; one has not completed a task since start-up
4Threads recover after five seconds, transfers fail under load, no hang
The four failures, and what fixes each
Each incident, the failure, and the fix
FailureFix
1DeadlockA global lock ordering, applied everywhere both locks are taken
2LivelockRandomised backoff, or one thread that decides rather than both yielding
3StarvationA fair lock, or work-stealing so no thread depends on winning a race
4Deadlock, with a timeout on topOrder the locks; keep the timeout as the net

Processor use separates the first two, which is why it is the first thing to look at. Idle and stuck is a cycle of locks; busy and stuck is threads reacting to each other.

The fourth is the one that gets closed as a different bug. Intermittent failures under load with no hang looks like a capacity problem, and it is the same cycle with a net under it.