Debug a container that will not start

A container that exits immediately gives you one line of output and no shell to poke at. The routine is always the same four commands, and the exit code usually names the cause before you read anything.

1. The routine

mkdir -p ~/labs/docker-debug && cd ~/labs/docker-debug

For any container that will not stay up:

docker ps -a --filter name=X              # what status and exit code
docker logs X                             # what it said before dying
docker inspect X --format '{{json .State}}'   # OOM? which signal?
docker run --entrypoint sh -it IMAGE      # get in without running the app

That last one is the trick people miss: override the entrypoint and you get a shell in the exact image, with none of the application running.

2. Failure one: the command does not exist

docker run --name f1 alpine:3.20 /usr/bin/myapp
docker ps -a --filter name=f1 --format '{{.Status}}'
docker inspect f1 --format 'exit={{.State.ExitCode}} err={{.State.Error}}'

Exit 127 — "command not found", the same convention as your shell. Almost always a wrong path in CMD, or a binary that was built in one stage of a multi-stage build and not copied into the final one.

3. Failure two: it is not executable

cat > start.sh <<'SH'
#!/bin/sh
echo "starting"
SH
cat > Dockerfile.f2 <<'DOCKER'
FROM alpine:3.20
COPY start.sh /start.sh
CMD ["/start.sh"]
DOCKER
docker build -q -f Dockerfile.f2 -t f2img . >/dev/null
docker run --name f2 f2img
docker inspect f2 --format 'exit={{.State.ExitCode}}'

Exit 126 — found but not executable. COPY preserves the host's mode, and a fresh file has no x bit. Fix it in the Dockerfile rather than relying on the host:

cat > Dockerfile.f2 <<'DOCKER'
FROM alpine:3.20
COPY --chmod=755 start.sh /start.sh
CMD ["/start.sh"]
DOCKER
docker build -q -f Dockerfile.f2 -t f2img . >/dev/null
docker run --rm f2img

4. Failure three: the config it needs is not there

cat > Dockerfile.f3 <<'DOCKER'
FROM alpine:3.20
CMD ["sh", "-c", "cat /etc/myapp/config.ini && echo started"]
DOCKER
docker build -q -f Dockerfile.f3 -t f3img . >/dev/null
docker run --name f3 f3img
docker logs f3
docker inspect f3 --format 'exit={{.State.ExitCode}}'

Exit 1 with a message — the useful case, where the application told you. Now get inside the image to check what _is_ there:

docker run --rm --entrypoint sh f3img \
  -c 'ls -la /etc | head -n 5; ls /etc/myapp 2>&1'

The entrypoint override is what lets you inspect an image whose application cannot start. docker exec cannot help here — there is no running container to exec into.

5. Failure four: killed rather than crashed

docker run -d --memory=32m --memory-swap=32m --name f4 alpine:3.20 \
  sh -c 'x=""; while :; do x="$x$(head -c 500000 /dev/zero | tr "\0" "a")"; done'
sleep 8
docker inspect f4 --format 'exit={{.State.ExitCode}} oom={{.State.OOMKilled}}'
docker logs f4 | tail -n 2

Exit 137 and empty logs. The signal-kill family is worth memorising because these produce _no_ application output:

6. Failure five: it exits 0 and you did not expect it to

docker run -d --name f5 nginx:alpine nginx
sleep 2
docker inspect f5 --format 'exit={{.State.ExitCode}} status={{.State.Status}}'
docker run -d --name f5b nginx:alpine nginx -g 'daemon off;'
sleep 2
docker inspect f5b --format 'exit={{.State.ExitCode}} status={{.State.Status}}'
docker rm -f f5b >/dev/null

The first exits cleanly: nginx daemonised, the foreground process returned, and a container lives exactly as long as PID 1. The second stays up because daemon off keeps it in the foreground.

Any process that backgrounds itself does this. There is nothing to debug in the logs — the container did what it was told.

7. ENTRYPOINT versus CMD, which explains a lot of this

cat > Dockerfile.ec <<'DOCKER'
FROM alpine:3.20
ENTRYPOINT ["echo", "entrypoint:"]
CMD ["default-arg"]
DOCKER
docker build -q -f Dockerfile.ec -t ecimg . >/dev/null
docker run --rm ecimg
docker run --rm ecimg replaced-arg
docker run --rm --entrypoint echo ecimg only-this

CMD is the default arguments, replaced by anything you put after the image name. ENTRYPOINT is the command, replaced only by --entrypoint. Confusing the two is why docker run myimage bash sometimes gives a shell and sometimes passes bash as an argument to the app.

8. When it is not the container

docker events --since 5m --until now --filter type=container 2>/dev/null | tail -n 5
docker inspect f4 --format '{{.RestartCount}} restarts'
sudo journalctl -u docker -n 10 --no-pager 2>/dev/null | tail -n 3

docker events is the timeline: creates, starts, dies, OOMs, health status changes, in order. A container restarting in a loop shows here as a repeating pattern, and RestartCount confirms it.

Clean up

docker rm -f f1 f2 f3 f4 f5 2>/dev/null
docker image rm f2img f3img ecimg 2>/dev/null
cd ~ && rm -rf ~/labs/docker-debug

Where this goes next

You can diagnose a container that will not start. Next: the flags that should be on every container you run in production.