Who this is for: Network engineers, SREs, red-teamers, SOC analysts, performance-tuning gurus, and senior developers who want a hands-on, no-nonsense field manual that scales from a Raspberry Pi lab to multi-continent SD-WAN backbones.
Network troubleshooting is the disciplined, evidence-driven workflow for detecting, isolating, and fixing data-path failures across every OSI/TCP-IP layer. It has two hard business KPIs:
A strong practice shrinks both, documents root cause, and feeds the lessons back into architecture, monitoring, and runbooks.
Reactive vs proactive: Reactive work stops fires; proactive work prevents them. Your tooling, metrics, and chaos drills must support both.
/24, /27, /31—why oddly sized masks matter for point-to-point links.ipcalc:ipcalc 192.168.14.0/29
| Check | Typical Command | What Success Looks Like |
|---|---|---|
| Link-lights & negotiation | ethtool eth0 |
1 G Full, no errors |
| Loopback plug | swconfig dev switch0 set loopback 1 |
Clean Rx/Tx counters |
| Optics power | ethtool -m eth2 |
Rx-Power within spec –-1 dBm to –3 dBm |
watch -n2 "ip -s link show eth0 | grep -A1 RX"
| Tool | Layer | Snippet | Insight |
|---|---|---|---|
ping -M do -s1472 dst |
3 | Path-MTU discovery | |
traceroute -I -T dst |
3 | Hop latency, MPLS labels | |
ip -s link |
2/3 | Errors, drops, speed | |
dig +trace fqdn |
7 | Delegation tree | |
ss -tulpn |
4 | Listening/ESTAB sockets | |
ip route get 8.8.8.8 |
3 | Chosen egress path | |
tcpdump -ni any 'tcp[13]&2!=0' |
2-7 | SYN flood health | |
nmap -sS -Pn -p1-1024 dst |
3-7 | Port open/filter | |
arp -a |
2 | Duplicate MACs | |
mtr -ezbwrc 100 dst |
3 | Real-time loss/latency |
show spanning-tree detail | include role – look for root inconsistent.curl -6 https://example vs curl -4 ….Idle → Active → OpenSent loops indicate auth/TTL problem.ip route show vrf red 0.0.0.0/0 must not appear in vrf blue.sequenceDiagram
Client->>Server: SYN
Server-->>Client: SYN-ACK ❌ (dropped)
Client->>Server: SYN (retries)
Usually firewall state-table exhaustion or asymmetric route.
sudo ethtool -k eth0 | grep offload.dig +dnssec +multi example.com — look for ad flag.curl -v https://site | grep HTTP — 499 vs 504 semantics.openssl s_client -servername site -connect ip:443 — verify SNI CN match.| Category | Symptom | Root Cause | Remediation |
|---|---|---|---|
| DNS | Long FQDN resolve | SERVFAIL from upstream | Fix zone-transfer ACL, bump SOA serial |
| Routing | Intermittent reachability | ECMP hash imbalance | Enable L4 hash, or pin flow with policy |
| Firewall | Random HTTPS resets | Shadow DROP above ACCEPT | Reorder rules, add logging prefix |
| Performance | 200 ms spikes | Bufferbloat on CPE | Apply FQ-CoDel: tc qdisc … fq_codel |
| MTU | TLS fails after 14 kB | ICMP black-hole | MSS-clamp: iptables --clamp-mss-to-pmtu |
mmcli -m 0 --command='AT+QENG="servingcell"'.# Trace path across Cilium overlay
cilium monitor --icmp --related -v
flannel.1 interface encaps.calicoctl node status to verify peer state.Mermaid graph of inbound/outbound:
graph TD
Client -->|mTLS| Envoy_Sidecar
Envoy_Sidecar -->|mTLS| App_Pod
App_Pod --> Envoy_Sidecar
Envoy_Sidecar -->|mTLS| Remote_Envoy
gcloud logging read.tcpdump -ni underlay udp port 4789.show crypto isakmp sa for phase-1 timers.ip netmask 255.255.255.0.zeek -i eth0 local "Site::local_nets += { 10.0.0.0/8 }"
Correlate notice.log with Suricata eve.json for context-rich alerts.
sysctl net.ipv4.tcp_congestion_control=bbr.ss -ti.tc qdisc add dev eth0 root handle 1: htb default 20
tc class add dev eth0 parent 1: classid 1:20 htb rate 10mbit ceil 20mbit
Enable WRED on class 1:20 for prioritized drops.
dig +short CHAOS TXT id.server @resolver to geolocate DNS POP.import http from 'k6/http';
export default function () {
http.get('https://api.example.com/health', { timeout: '2s' });
}
Run hourly via Kubernetes CronJob; raise PagerDuty on P95 > 300 ms.
| Stack | Open-Source | Commercial |
|---|---|---|
| NPM | LibreNMS, Prometheus, Grafana | SolarWinds, PRTG |
| AIOps | Zabbix + Python ML | Kentik, ThousandEyes |
| Packet Capture | Wireshark, Arkime | Gigamon GigaVUE |
| APM | OpenTelemetry | Datadog NPM, New Relic |
route-map.ip rule 100 due to Cilium bug.cilium bpf ct flush, cordon & drain, daemonset restart → restored.Operational discipline plus the right depth of packet-level insight turns firefighting into a repeatable science—keeping latency low, throughput high, and users happy.
# MTU discovery (fails on DF exceed)
ping -M do -s 1472 8.8.8.8
# Real-time TCP retransmissions
tcpdump -ni any 'tcp[13] & 0x10 != 0 and tcp[13] & 0x08 != 0'
# Show route advertisement (Juniper)
show route advertising-protocol bgp 192.0.2.1
# Map Kubernetes VIP to endpoints
kubectl get ep kube-dns -o wide
TCP Flags: URG ACK PSH RST SYN FIN
IPv6 Ext Headers: 0 Hop-by-Hop | 43 Routing | 44 Fragment | 50 ESP | 51 AH
DNS Opcodes: 0 QUERY | 5 UPDATE | 4 NOTIFY
| Data Type | Hot Storage | Cold Storage | Compliance |
|---|---|---|---|
| Raw pcap | 7 days SSD | 30 days S3/Glacier | PCI-DSS |
| Flow/metrics | 13 months TSDB | 2 years object store | GDPR |
| Syslog/audit | 1 year | 5 years tape | HIPAA |
If you found this content valuable, imagine what you could achieve with our comprehensive 47-week elite training program. Join 1,200+ students who've transformed their careers with Unit 8200 techniques.