Kubernetes pod liveness

An old joke goes:

A doctor gets a phone call from his best friend, and the frantic voice at the other end says “God, oh God, my wife’s dead! The doc tells his friend to calm down. “OK, now the first thing is, you have to be sure that she’s really dead.” He hears silence at the other end, then a single gunshot. “OK, that did it. What next?”

That’s actually quite similar to how kubernetes liveness checks works.

A liveness-check means that every now and then kubernetes will ask the pod if it is alive. And if it is not alive, kubernetes will kill the pod. The scheduler will then make sure another one is restarted.

That may sound good in theory. However let’s say your app is not really dead, it’s just not able to answer kubernetes fast enough. Maybe it’s just overwhelmed. When the app is overwhelmed is exactly when you don’t want to restart it.

So in conclusion, kubernetes liveness checks can make “my app is slow” into “my app is down”.