Istio Pilot - Debugging and Development workflow

As I've been touching more on Istio Pilot codebase recently, I've found very difficult to debug it without a DLP/debugger connected to it.

Usually I would say "just add some fmt.Prints and spew.Sdump" around the code, but given I need to understand step by step how Istio converts some Gateway API resources into its own model, attaching a debugger became a required step.

Additionally, the "compile/push/debug/repeat" loop became essential, so I kind of did my own workflow, that works for me.

All of my development workflow is done using my Gateway API fast provisioning script on kind.

Note: This is my development workflow, and may not work for you, so take with with a grain of salt ;)

Compile/Push repeat

This is my copy/paste/be happy script:

export TAG=istio-$(date +%s)

export HUB=localdev.tld/istio

export DEBUG=1

make docker.pilot && kind load docker-image ${HUB}/pilot:${TAG} &&  kubectl set image -n istio-system deployments/istiod discovery=${HUB}/pilot:${TAG}

Remote debugging

First, why remote debugging? I wanted to check exactly what's going on my live code. I didn't wanted to run it locally.

I will update this article with a better approach, but for now:

• Create and apply a patch on Dockerfile.pilot AND REMEMBER TO NOT COMMIT IT!
  

This patch gets delve from the Go official images, and copy to the final Istio image. It also changes the entrypoint to be delve starting pilot.

curl -o dockerfile.patch https://gist.githubusercontent.com/rikatz/f5a07945038f705c9d10625eceb99932/raw/c56c9f0fca2545fffbe87d4de55ccb5d474f173b/dockerfile.patch

git apply dockerfile.patch

Note: right now bumping the used go image should be done manually .

• Change your Pilot deployment to allow writing to filesystem and to add SYS_PTRACE capability

These changes are required for Delve to work properly

 kubectl patch deployment istiod -n istio-system --type='json' -p='[

    {"op": "replace", "path": "/spec/template/spec/containers/0/securityContext/allowPrivilegeEscalation", "value": true},

    {"op": "replace", "path": "/spec/template/spec/containers/0/securityContext/readOnlyRootFilesystem", "value": false},

    {"op": "add", "path": "/spec/template/spec/containers/0/securityContext/capabilities", "value": {"add": ["SYS_PTRACE"]}}

]'

• Do a port-forward to delve port

This is the port where you will connect your remote-debugger later (on localhost:40000):

kubectl port-forward -n istio-system deploy/istiod 40000

• Attach to remote process
  Here is my vscode file:
  

{

    "version": "0.2.0",

    "configurations": [

        {

            "name": "Remote Debug Pilot in K8s",

            "type": "go",

            "request": "attach",

            "mode": "remote",

            "remotePath": "",

            "port": 40000,

            "host": "localhost",

            "showLog": true,

            "trace": "verbose",

            "substitutePath": [

                {

                    "from": "${workspaceFolder}",

                    "to": "/work"

                }

            ]

        }

    ]

}

Running integration tests

Finally, this is my last step. Once I am writing some integration tests, I may want to run them locally before posting my PR. Here is usually how I do:

INTEGRATION_TEST_FLAGS="-run ^TestGatewayAPIResourcesOnly$"   ./prow/integ-suite-kind.sh --topology MULTICLUSTER  --skip-cleanup   test.integration.pilot.resourcefilter.kube

This one runs a single integration test called TestGatewayAPIResourcesOnly. The topology flag is optional and can be used if a specific topology integration test is failing.

The final name test.integration.pilot.resourcefilter.kube can be translated to "tests/integration/pilot/somesubfolder".

So, in case you want to run a test that is directly on "tests/integration/pilot" you can replace it for test.integration.pilot.kube.