🔧 Remote debugging Go App

For the longest time, I wasn’t a fan of debuggers. Coming from a background in Spring Framework, Java, Python, and PHP (Symfony/Laravel), I always found logs and traces more reliable for debugging. I had even dabbled with GDB in the early days, but it didn’t stick. Instead, I relied on logging to figure out what my applications were doing.

However, my perspective changed when I started working with Go. While developing in Kubernetes and working on microservices, I faced a situation where the complexity of the service made logging and tests insufficient. This was a large service with multiple dependencies, serving both users and operators, making bug fixes particularly challenging.

Let me share a recent experience where using a debugger saved me a significant amount of time.

Debugging in Kubernetes: A Real-World Example

A few weeks ago, I was developing a feature that affected multiple parts of a microservice running in Kubernetes. This service was being called by several other components (like the app and admin panels), and I needed to trace what was happening inside the service, step by step. Tests couldn’t cover everything, especially when I needed to ensure that certain flags were set correctly to prevent sending multiple SMS notifications to users.

I could have used logs to trace the service, but every time I missed an entry, I would have had to add a new one, push the code, and wait for the CI pipeline to complete—wasting 3-5 minutes each time just to deploy to staging.

Instead, I decided to try using a debugger. Here’s how I set it up.

Setting Up Delve Debugger in a Kubernetes Pod

After a lot of trial and error, I found that adding the following lines to the Dockerfile allowed me to run Delve (a debugger for Go) inside the Kubernetes Pod:

ENTRYPOINT ["/bin/sh", "-c", "/go/bin/dlv --listen=127.0.0.1:8001 --headless=true --api-version=2 --only-same-user=false exec /path/to/exec"]

To install Delve, I added this line to the Dockerfile:

RUN go install github.com/go-delve/delve/cmd/[email protected]

Next, I compiled the Go application with specific flags to disable optimizations and inlining:

go build -gcflags "all=-N -l"

Connecting to the Debugger

Once the service was deployed, I used port forwarding to connect to the debugger. I added the following port configuration to the Kubernetes manifest:

ports:
  - name: dlv
    containerPort: 8081

Then, I forwarded the port to my local machine:

oc port-forward $(oc get pods -l app=LABEL -o name | head -n 1) 8001:8001

Now I could connect to the debugger from my local machine using:

dlv connect :8001

Setting Breakpoints and Configuring Paths

With the debugger connected, I set breakpoints, such as:

(dlv) break main.main

Because the service was built in a different environment (GitLab’s pipeline), the paths didn’t match my local setup. To solve this, I used substitute-path to map the paths:

(dlv) config substitute-path /usr/local/go/ /Users/hossein/sdk/go1.23.1/

Running these commands every time was tedious, so I created a script (dlv.init) to automate the setup:

dlv connect :8001 --init dlv.init

Explanation of Key Commands

Delve Debugger Command:

dlv --listen=127.0.0.1:8001 --headless=true --api-version=2 --only-same-user=false exec /path/to/exec

• --listen: Opens the debugger on 127.0.0.1:8001 for remote connections.
• --headless=true: Runs Delve without an interactive UI, suitable for remote debugging.
• --api-version=2: Uses API version 2 for better tool compatibility.
• --only-same-user=false: Allows users other than the process owner to connect.
• exec /path/to/exec: Starts or attaches to the Go executable for debugging.

This setup allows remote debugging of a Go service in a Kubernetes environment.

Go Build Command:

go build -gcflags "all=-N -l"

• -N: Disables optimizations.
• -l: Prevents inlining of functions.

These flags ensure the code stays closer to the source, making it easier to debug by allowing you to step through every function.

Example Usage

To demonstrate how to debug a remote service, I use a local Docker container for simplicity. However, in practice, this service would be deployed on Kubernetes.

To build the Docker image, run the following command:

make build

This command will build a Docker image. Once built, you can start the service by running:

make start

Output:

$ make start
docker run -it -p 8001:8001 -p 8080:8080 --rm my-app
API server listening at: [::]:8001
2024-10-12T13:48:31Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)

The prompt shows that the debugger is running. To attach the debugger to the remote service, run:

make debug

Output:

$ make debug
bash -c "dlv connect :8001 --init <(sed 's|PWD|'`pwd`'|g; s|HOME|'OME'|g' dlv.init)"
Type 'help' for list of commands.
Breakpoint 1 set at 0x2188cc for main.main.func1() ./main.go:9
(dlv) c

Once attached, send the c command to continue execution. Then, send a request to the service using the following:

make send-request

The debugger will show output like this:

> [Breakpoint 1] main.main.func1() ./main.go:9 (hits goroutine(17):1 total:1) (PC: 0x2188cc)
Warning: debugging optimized function
Warning: listing may not match stale executable
     4:     "fmt"
     5:     "net/http"
     6: )
     7:
     8: func main() {
=>   9:     http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    10:         fmt.Fprintf(w, "Hello, World!")
    11:     })
    12:
    13:     http.ListenAndServe(":8080", nil)
    14: }

At this point, you can manually debug the service. For example, to inspect the HTTP method, use:

(dlv) p r.Method
"GET"

Once troubleshooting is complete, use the c command to continue and the client will receive the output.

Conclusion

Using a debugger in this case allowed me to step through the code and understand the service's behavior without constantly pushing new logging code. It was a game-changer, especially when working with complex microservices in Kubernetes. Debugging Go in production environments can be daunting, but with the right setup, it can save you a lot of time and frustration.

You can check out the full example of this setup on my GitHub.