Skip to content
Linux Administration
Lab 24 of 27·40mIntermediate

Debug a network path from one end

Take "it times out" apart into six separate questions — DNS, route, listener, firewall, TLS, and the application — and answer each with one command.

You need

  • A Linux VM with sudo access and internet access
  • iproute2, dnsutils, curl, tcpdump (apt-get install -y iproute2 dnsutils curl tcpdump)

Do first

"The service is unreachable" is not one problem. It is six, in a fixed order, and testing them out of order is how an afternoon disappears. Work up the stack.

1. Does the name resolve

dig +short example.com
dig example.com | sed -n '/ANSWER SECTION/,/^$/p'
resolvectl status 2>/dev/null | head -n 12 || cat /etc/resolv.conf

dig queries DNS directly and ignores /etc/hosts; getent hosts example.com uses the full resolver path the way your application will, including /etc/hosts and nsswitch. When those two disagree, you have found the bug.

Verify

getent hosts example.com | wc -l # 1 or more

2. Which interface and route would be used

ip -brief address
ip route get 1.1.1.1
ip route show default

ip route get is the precise question: given this destination, which source address, interface and gateway does the kernel pick? Far better than reading the whole table and reasoning about longest-prefix matches yourself.

Verify

ip route get 1.1.1.1 | head -n 1 # 1.1.1.1 via <gateway> dev <iface> src <your ip>

3. Is anything listening, and on which address

sudo ss -tlnp
sudo ss -tlnp 'sport = :22'

Read the Local Address column carefully. 0.0.0.0:22 accepts from anywhere; 127.0.0.1:22 accepts only from the machine itself. A service bound to loopback is the most common cause of "works when I curl it on the box, times out from outside" — and no amount of firewall work will fix it.

Verify

sudo ss -tln | grep -E ':22\s' | head -n 1 # a LISTEN row for port 22

4. Distinguish refused from filtered

Start a listener and talk to it:

(nc -l -p 9999 &) 2>/dev/null || (ncat -l 9999 &) 2>/dev/null
sleep 1
curl -s -m 3 -o /dev/null http://127.0.0.1:9999
echo "open port exit=$?"
curl -m 3 http://127.0.0.1:9998; echo "closed port exit=$?"

The distinction matters more than either result:

  • Connection refused, immediately: the packet reached the host and nothing was listening. Routing and firewall are fine; the service is down.
  • Timeout, after seconds of nothing: something dropped the packet silently. That is a firewall, a security group, or a route — not the service.

Refused is good news. Timeouts are where you go looking at network ACLs.

Verify

curl -m 3 http://127.0.0.1:9998 2>&1 | grep -o "Connection refused" # Connection refused

5. Read the local firewall

sudo iptables -L -n -v 2>/dev/null | head -n 15
sudo nft list ruleset 2>/dev/null | head -n 20
sudo ufw status verbose 2>/dev/null

Three tools, one kernel subsystem. Modern systems use nftables with iptables as a compatibility shim, so both commands may show the same rules described differently. On a cloud instance, remember the host firewall is only half — the provider's security group drops packets before they ever arrive, and nothing on the box can see that happening.

Verify

sudo nft list ruleset >/dev/null 2>&1 || sudo iptables -L -n >/dev/null 2>&1; echo "firewall readable: exit=$?" # exit=0

6. Watch the actual packets

When the layers above all look correct and it still fails, stop reasoning and observe:

sudo timeout 10 tcpdump -n -i any 'tcp port 443 and host example.com' &
sleep 1
curl -s -o /dev/null https://example.com
wait

Read the handshake: S is your SYN, S. is the SYN-ACK, . is your ACK. What you see tells you where it stopped.

  • SYN with no reply: dropped outbound, or the reply is dropped. Firewall or route.
  • SYN then RST (R): actively refused.
  • Full handshake, then nothing: the network is fine and the application is not answering.

Verify

sudo timeout 8 tcpdump -n -c 3 -i any 'tcp port 443' 2>/dev/null | wc -l # 3 or more packet lines while something is talking HTTPS

7. TLS and the application, separately

curl -sv --max-time 10 https://example.com -o /dev/null 2>&1 \
  | grep -E "SSL|subject|issuer|expire|HTTP/"
echo | openssl s_client -connect example.com:443 \
  -servername example.com 2>/dev/null \
  | openssl x509 -noout -subject -dates

openssl s_client tests TLS with no HTTP involved, which separates a certificate problem from an application problem. The -servername flag sets SNI — omit it against a host serving many sites and you will get the wrong certificate and a confusing error.

Verify

echo | openssl s_client -connect example.com:443 \ -servername example.com 2>/dev/null \ | openssl x509 -noout -checkend 86400 \ && echo "cert valid for at least a day" # cert valid for at least a day

The order, as a checklist

  1. getent hosts — does the name resolve the way the app resolves it
  2. ip route get — which interface and gateway
  3. ss -tlnp — is it listening, and on which address
  4. refused vs timeout — service down, or packets dropped
  5. nft/iptables plus the cloud security group
  6. tcpdump — where the handshake actually stops
  7. openssl s_client — TLS separately from the application

Clean up

pkill -f "nc -l" 2>/dev/null; pkill -f "ncat -l" 2>/dev/null; echo done

Where this goes next

You can locate a failure in a network path. The last guided lab is about a box that is up, reachable, and simply too slow.