Posted on 6 January 2020, updated on 21 September 2023.
Debugging a Kubernetes environment can get tedious if you do not know the right commands to quickly deploy the tools you need at the right time. The purpose of this article is to provide some commands, tools, and practices that can help you debug faster in a Kubernetes context.
Looking at Kubernetes error messages
You can start debugging by looking for error messages. In Kubernetes, they will often take the form of a resource called "event" that you can list with the command `kubectl get events`. It provides a centralized view of events in kubernetes like pods entering a “Crashloop Backoff” state or “OOM killed”.
Looking at Kubernetes pods logs
Then you can check Kubernetes pods logs running `kubectl logs <pod-name> -f` but it forces you to put the exact pod name and you can not aggregate logs from several pods.
Stern allows you to aggregate logs of all pods and uses a regular expression to filter them `stern <expression>`
Debug flux opening
You might want to quickly check if a route is opened. To do so, run and ssh in a busybox minimal bare operating system in a single command `kubectl run --generator=run-pod/v1 -i --tty busybox --image=busybox --restart=Never -- sh`. It contains several useful tools for debugging.
Debug services connections
For an http call the same command as above with an image provided with curl will do `kubectl run --generator=run-pod/v1 -i --tty busybox --image=radial/busyboxplus:curl --restart=Never -- sh`
Debug databases connections
You might want to quickly test the connection to a database. Each database has its client and each client works differently but you can always run them quickly. For instance, get a mysql client in your kubernetes cluster with the following command
`kubectl run mysql -it --rm --image=mysql -- mysql -h <ip> -P <port> -u <user> -p<password>`
Similar commands exist for postgres, sqlserver, oracle ...
Low level debug with kube-debug
Finally, for debugging lower level Linux stuff in a kubernetes pod, you can use kubectl-debug. It will inject a container that will share pid, network, user and ipc with other containers of the pod. Also, adding the `--fork` option allows you to fork pods that are in crashloop backoff so that you can still debug them. It is pretty neat!
Here are the tools and commands we use to debug in a Kubernetes environment, tell us yours in the comment section! Also check our other articles on Kubernetes.