Mastering strace: Comprehensive Guide to Network Packet Analysis with tcpdump, Wireshark, and More

1 strace

2 tcpdump

3 netstat

4 ifconfig

5 iptraf

6 iftop

7 sar

8 wireshark

1 strace: trace system calls and signals

strace -p 28182 -s4086 -T -f [-e trace=network/process/signal]

2 strace: Trace system calls and signals

General output information of a TCP packet captured by TCPDUMP is:

src > dst: flags data-seqno ack window urgent options length

src > dst: indicates the source to destination address, flags are the flag information in TCP packets; S is the SYN flag, F is FIN, P is PUSH, R is RST, “.” means no flag; data-seqno is the sequence number of the data in the packet, ack is the expected next sequence number, window is the size of the receive buffer window, urgent indicates whether there is an urgent pointer in the packet. Options are options. Length is length.

UDP packet output information

route.port1 > ice.port2: udp length

A UDP packet sent from the port1 of host ROUTE to port2 of host ICE, the type is UDP, the length of the packet is length.

Common parameters

-i specifies the network card to capture packets

-nn disables tcpdump from displaying IPs, ports, etc., and converting them to domain names and well-known service names

-s specifies the packet size to capture. Use -s 0 to specify a packet size of 262144 bytes

-c specifies the number of packets to capture

-w parameter specifies saving captured packets to a file

-X excludes link layer frames, displaying in ASCII encoding (tcpdump can show in hexadecimal and ASCII format)

-A prints packet contents except for link layer frame headers in ASCII encoding

-d provides matching packet code in a user-readable assembly format

-dd provides matching packet code in C language code segment format

-ddd provides matching packet code in decimal format

tcpdump tcp -i eth1 -t -s 0 -c 100 -X and dst port ! 22 and src net 192.168.1.0/24 -w ./target.cap

tcpdump -iany -s0 port 80 -A

[Instance troubleshooting: Testing the connectivity of high-defense IP]

– We have an external machine A: 49.51.39.154

– Have the user capture packets on the server: tcpdump -iany -s0 src host 49.51.39.154 -A

– We execute on machine A: telnet xxx.gsadds.com xxx

– If the client machine can display our request data, connectivity is good:

1. root@VM_127_231_centos ~]# tcpdump -iany -s0 src host 49.51.39.154 -A
2. tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
3. listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
4. 11:44:42.851203 IP 49.51.39.154.53440 > VM_127_231_centos.21150: Flags [P.], seq 2707990323:2707990332, ack 193352084, win 229, options [nop,nop,TS val 855746637 ecr 2046056509], length 9
5. Eh.=.'@.8...13'.
6. .....R..h.3..Q.....Y......
7. 3..My.X=lippman
8. ................

Filter rules

host a.b.c.d specifies capturing only data communication between the host and another host a.b.c.d

tcp port x: specifies capturing only data communication with TCP protocol destination port or source port as x

icmp: specifies capturing only data communication with ICMP protocol.

!: reverse match, for example, port ! 22, captures data communication not port 22.

Can be combined using and or or

3 netstat

Netstat -apn

netstat -i

4 iptraf: Interactive Colorful IP LAN Monitor

5 iftop

6 7 sar Collect, report, or save system activity information

7 Packet data analysis – Wireshark packet warning messages

1. [Packet size limited during capture]

The marked packet was not completely captured, usually due to the capture method, e.g., tcpdump with the -s option specified too small

2. [TCP Previous segment not captured

In TCP transmission, data segments sent from the same host should be continuous, i.e., the sequence number of the next packet equals the previous packet’s Seq + Len (the three-way handshake and four-way wave do not count). If Wireshark finds the next packet’s Seq number greater than the previous packet’s Seq + Len, it knows part of the data in between is missing. There are two cases for a network packet not being captured: one is it really got lost; the other is it’s not really lost but the capture tool missed it. How to distinguish between these two in Wireshark? Just look at the other side’s acknowledgment (Ack). If the acknowledgment includes the packet that wasn’t captured, then the capture tool missed it, otherwise it really was lost

3. [TCP ACKed unseen segment]

Wireshark will show a warning message when it identifies an acknowledged packet wasn’t captured

4. [TCP Out-of-Order]

During TCP transmission (excluding the three-way handshake and four-way wave), data packets sent from the same host should be continuous, i.e., the sequence number of the succeeding packet equals the preceding packet’s Seq + Len. One can also say the succeeding packet’s Seq will be greater than or equal to the preceding packet’s Seq. When Wireshark finds the succeeding packet’s Seq number less than the preceding packet’s Seq + Len, it considers it as out-of-order, hence the [TCP Out-of-Order] warning. Small-scale out-of-order does not matter much, for example, the order 1, 2, 3, 4, 5 swapped to 2, 1, 3, 4, 5 is ok. But if the disordering is large enough, fast retransmission could be triggered, for example, swapping to 2, 3, 4, 5, 1 can lead to generating enough Dup ACKs to cause the 1st packet’s retransmission.

5. [TCP Dup ACK]

When disordering or packet loss occurs, some packets with Seq numbers larger than expected are received by the receiver. For each such packet received, it will Ack the expected Seq number, alerting the sender in this way, thus generating some duplicate ACKs. Wireshark marks these duplicate ACKs with [TCP Dup ACK] .

6. [TCP Fast Retransmission]

When the sender receives 3 or more [TCP Dup ACK], it notices that the previously sent packets may have been lost and thus quickly retransmits them (this is an RFC stipulation)

7. [TCP Retransmission]

If a packet is indeed lost, and there are no subsequent packets to trigger [Dup Ack] at the receiver, it won’t be quickly retransmitted. The sender will then have to wait until timeout to retransmit, and such retransmitted packets will be marked as [TCP Retransmission] by Wireshark.

1. [TCP CHECKSUM INCORRECT] Usually caused by a TCP checksum offload issue, (use network card hardware to calculate the checksum instead of letting the TCP/IP stack calculate it)