Skip to content

How to troubleshoot using the namespace of a container

This document (000020163) is provided subject to the disclaimer at the end of this document.

Environment

RKE2 environment with containerd and RKE environment with docker

Situation

When troubleshooting an issue, often a faithful reproduction and an exact environment are needed. This can be a challenge in a containerized environment, where tools and a shell environment may not be easily available within containers of a Pod.

With more images going distroless, container images often don't have the necessary tools such as `dig`, `curl`, `telnet`, etc... since they not included. This can make troubleshooting a pod's networking connections very difficult, especially in an air-gapped environment.

Resolution

RKE2 Environment

With crictl and nsenter, the node's OS packages can be utilised to troubleshoot the pod container's networking by entering the container's network namespace without using kubectl exec.

Setup

  • Get container ID(s) from a pod
POD_NAME=<pod name>
NAMESPACE=<namespace for pod>

kubectl describe pod -n $NAMESPACE $POD_NAME | awk -F'//' '/containerd:/ {print $2}'
  • Copy and save the container ID for later. Next, identify the node that the pod is on
kubectl get pod -n $NAMESPACE $POD_NAME -o wide

If the node in the output is different from your current node, be sure to switch to a shell session on the corresponding node.

Using crictl

When using critcl in an RKE2 setup, you may need to run the following commands before using crictl

export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
PATH="$PATH:/var/lib/rancher/rke2/bin"
  • From the node where the pod is deployed, run the following command to get the PID of a container from the setup steps
CONTAINER_ID=<container ID>
PID=$(crictl inspect -o go-template --template '{{.info.pid}}' $CONTAINER_ID)

nsenter

  • Run commands from the node within the network namespace context of the container/Pod with nsenter
nsenter -n -t $PID <command>
  • Run commands from the node within the context of all of the container/Pod namespaces with nsenter
nsenter -a -t $PID <command>

RKE Environment

Two approaches can be taken:

Sidecar Container

By running a container in the same namespaces as another, it's possible to use that container for troubleshooting.

The sidecar container can be started using the same network and PID namespaces while attaching the same volumes:

  • Set the ID or name of the container you wish to troubleshoot:
ID=<container ID or name>
  • Run the sidecar container using the network, PID and volumes
docker run -it --net=container:$ID --pid=container:$ID --volumes-from=$ID alpine sh

It is now possible to troubleshoot with commands from the alpine container, within the context of the container or Pod with the issue.

For example, if you were experiencing a network issue from this Pod, it is now possible to use tools available in the sidecar container to simulate the connection, view the network configuration and troubleshoot interactively.

Substitute the alpine container as needed with an image of your choice.

Note, this will attach the same volumes as the parent container, but the parent container read/write layers will not be accesible - to access the same container filesystem, see the nsenter example below.

Use nsenter from the host

Alternatively, tools are available on the host for the same use case with the nsenter command. The nsenter command is standard on most Linux distributions, for example, on Ubuntu, it is provided by the util-linux package.

  • Set the ID or name of the container you wish to troubleshoot:
ID=<container ID or name>
  • Obtain the first process in the container (PID 1):
PID=$(docker inspect --format '{{ .State.Pid }}' $ID)
  • Run commands from the node within the network namespace context of the container/Pod with nsenter:
nsenter -n -t $PID <command>
  • Run commands from the node within the context of all of the container/Pod namespaces with nsenter:
nsenter -a -t $PID <command>

For example, when troubleshooting a network issue, tools like tcpdump, curl, dig, and mtr can be used to interactively troubleshoot the issue.

Note, the -a flag is available in recent versions of nsenter, if this does not succeed, use a flag for a specific namespace, check the nsenter --help output.

Status

Top Issue

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.