Problem framing
Containers are everywhere in dev chats right now. Docker got a big bump with new networking and volumes, CoreOS is pushing rkt, Kubernetes keeps getting steadier. We ship faster, but when something breaks inside a container, the fix often feels like guesswork. Logs give hints. Metrics point to symptoms. The root cause often sits inside a Linux namespace you have not looked into yet.
Here is the trick most folks skip on the first incident. Treat the container as a set of namespaces. There is a network namespace with its own routes and firewall rules. A mount namespace with its own view of the file system. A PID namespace with its own process tree and signal handling. When you step into the right namespace on the host, the mystery fades. Tools like nsenter turn a black box into regular Linux, and you can do real debugging of containers with confidence.
Three case walkthrough
Case 1: The container cannot reach a service
Symptom: your app talks fine on your laptop, but inside the container it times out. Sometimes DNS fails inside the container while the host resolves the same name just fine. Sometimes a published port looks open from outside, yet traffic never lands on the process.
Think network namespace. Find the container process on the host and step into its network view. From there, check the interface addresses, default route, and the resolver config the app is actually using. If the app runs in Kubernetes or through Docker’s bridge, verify the veth pair is attached and the bridge has the expected subnet. On the host, inspect iptables NAT rules for port publishing and outbound masquerade. If you run overlays like flannel or weave, confirm the peer routes exist and the control agent is healthy.
Common findings: DNS points to a server only visible on the host namespace, not inside the container. A missing default route after a restart. A policy rule dropped by a security agent. Port publish rules that only bind to localhost on the host. Once you inspect from inside the container’s network namespace, the cause tends to show itself in minutes.
Case 2: A volume looks empty or permissions go sideways
Symptom: the app expects a config file from a bind mount, but inside the container the path looks empty or read only. Or the app runs as a user that cannot read a file that exists on the host.
Think mount namespace. Step into it and list what the process really sees. Confirm the host path that was mounted, and watch for path typos and relative paths that resolve differently at runtime. If you are on CentOS or Fedora, look for SELinux labels on the host path that block the container. On Ubuntu, consider AppArmor profiles. When using overlay filesystems or devicemapper, remember that the container sees a copy on write view. A file you edited on the host may not be the same inode the process reads in its layer stack.
Common findings: the mount didn’t happen because the host path did not exist at start. SELinux denied writes to a bind mount without the right label option. The container root file system is read only by design and the app tries to write under system paths. Viewing the world from the container’s mount namespace makes these issues obvious.
Case 3: The container will not stop or leaves zombies
Symptom: you send a stop and it hangs. The process ignores signals. You see zombies piling up. CPU climbs even though the main process shows idle.
Think PID namespace and signal flow. Check the process tree the container actually runs. Many images start the app directly without a tiny init to reap children. That means zombies stick around and signals do not reach the right process group. Use a small init like tini or runit as PID 1 in the image, so children get reaped and signals flow. If the container refuses to die, look at open files and sockets from inside the PID namespace to spot a stuck worker. Also check the host logs for OOM killer messages if memory limits are tight, since a killed helper can leave the main process waiting forever.
Objections and replies
Objection: why not just use docker exec. Reply: exec is great when the engine is healthy and the container is responsive. When the runtime is wedged, when you need to inspect the network namespace of a pod, or when the process is already gone and you still need the mount namespace, nsenter on the host works every time.
Objection: stepping into namespaces sounds risky. Reply: only host root can do it, and you should log and audit that access anyway. For production, keep a small set of people with that power and document the playbook.
Objection: this feels too low level. Reply: it is simple once you try it, and it travels across Docker, rkt, LXC, and whatever we run next. Namespaces are the stable ground under all of this, which makes them perfect for container debugging.
Action oriented close
Make this a habit. When a container breaks, ask which namespace does this live in. If traffic fails, step into the network namespace and test names, routes, and port rules from there. If files look wrong, step into the mount namespace and confirm the exact paths and labels. If it will not die, step into the PID namespace and look at the process tree and open handles.
Build a small kit on the host. Keep nsenter, a way to inspect routes and sockets, a DNS client, and tools to look at processes and files. If you prefer, ship a tiny toolbox container you can join to the target namespace for repeatable checks. Use a lightweight init in your images so signals and zombies do not ruin your day.
Write down the steps your team will follow. Add a one pager to your runbook with the exact actions for network, mount, and PID cases. Add a note on SELinux or AppArmor if you use them. Add a line on where your NAT rules live. The next time you hear the words it works on my host, you can answer with a smile, step into the right namespace, and get the thing fixed.
If you care about debugging Docker containers without drama, learn to move across Linux namespaces. It is not fancy. It just works, and it will save you hours when the pager buzzes.