After writing How to develop Go gRPC microservices and deploy in Kubernetes, I received many queries about how to debug applications inside Kubernetes.
Deploying container, service or secret configuration is super easy in Kubernetes. But what about debugging? As a developer, it is always useful to be able to debug application with your favorite IDE.
When your application only works with Kubernetes API, you can simply launch your application from IDE and connect it to the remote Kubernetes API. But when your application needs to connect to other systems which are only available inside the Kubernetes cluster then this solution will not work.
For this article I will use kubernetes-go-grpc repository. You can find more details and deployment instructions in this article How to develop Go gRPC microservices and deploy in Kubernetes. As this article is focused on debugging solution, I am not going to describe about the project.
At the time generating binary go compiler can change the sequence of operations, add code, remove code or apply transformations. So, it is harder to map Go line to the optimized output. But during debugging you need to map or stop execution at a specific Go line. For this reason, you need to generate binary without optimization. You can disable optimization by passing flag to the compiler using
gcflags. Execute below command to generate binary without optimization
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -gcflags "all=-N -l" -o ./app
-N will disable optimization and
-l disable inlining. To remote you need to start a headless Delve server on the machine.
dlv --listen=:40000 --headless=true --api-version=2 --accept-multiclient exec ./app
Then connect to that server from your IDE (in this case Goalnd).
To containerize your application use below dockerfile named as
FROM golang RUN go get -u github.com/go-delve/delve/cmd/dlv RUN mkdir app WORKDIR /app COPY app . EXPOSE 40000 EXPOSE 8080 ENTRYPOINT ["/go/bin/dlv", "--listen=:40000", "--headless=true", "--api-version=2", "exec", "./app"]
Here download Delve for debugging, copy your debug enable binary, expose 8080 port as api is running on this port and 40000 port as Delve server will start on this port. To build your docker image execute
docker build -f Dockerfile.debug -t api-debug .
To use your local docker repository and deploy in kubernetes execute below command
eval $(minikube docker-env) \ && kubectl run --rm -i debug-api-deploy --image=api-debug --image-pull-policy=Never
To know your pod name execute
kubectl get pods
and find your api pod name. You pod name should be something like
debug-api-deploy-867bd55f4d-gzh5c. Now you need to forward port 8080 and 40000 so that you can connect your pod remotely.
kubectl port-forward <YOUR_POD_NAME> 40000:40000 kubectl port-forward <YOUR_POD_NAME> 8080:8080
Configure the IDE
In this article you will use Goland but it will work similarly with other IDE. You need to create a Remote Debug and point it to localhost:40000 since you have activated the port forward.
From that point, you are able to set Breakpoint in our local IDE, and it will communicate with the
Delve debugger deployed in the Kubernetes cluster to allows to debug your application.
The application will wait for your IDE to connect to it before starting execution of code, that way you are able to start debugging from start. Normally, the Pod will stop in the state completed each time you stop debugging on IDE side or loose connection.
Happy coding .... :)