TCP RST Packet Analysis: Understanding Connection Issues

The failure to establish a TCP connection often involves more than just a lack of response; it frequently entails the transmission of RST (Reset) packets.

Grasping the reasons for RST packet transmission is crucial for maintaining network stability and security. Through meticulous TCP RST packet analysis, network issues can be effectively diagnosed, network performance can be enhanced, and network security can be fortified.

The RST packet serves as a critical data packet for “connection reset” within the TCP protocol. In this article, we will provide a detailed analysis of TCP RST packets.

01 Understanding the Role of TCP RST Packets in Connection Failures?

TCP RST packet, that is, a data packet with the RST bit in the TCP header set to 1. According to RFC 793, the TCP RST packet is used to terminate an existing connection or reject a connection request.

When a TCP endpoint wishes to terminate a connection immediately, it sends a RST packet.

Therefore, when either the client or the server needs to reject the connection or believes that an error has occurred in the connection, a RST packet can be sent to reset the current TCP connection.

The embodiment of the RST packet in the TCP timing diagram is shown in Figure 1:

02 How to analyze the RST packet during the connection establishment phase?

During the connection establishment phase, the server may send a RST packet to reject the connection establishment. As shown in FIG. 2 , during the connection establishment phase, the server sends a RST packet, indicating that the server rejects the connection.

If the server sends a RST packet to reject the connection, it may be caused by the following reasons:

• The server port is not open: If the server does not run a network service on the port, or the service requested by the client on the server fails, the server will reject new connections through RST.

• The number of TCP connections on the server has reached the limit: If the server sets tcp_abort_on_overflow=1, the server will send a RST packet to reject the connection when the queue is full.
• Time_Wait state: If the current socket used by the client has just ended the previous connection and the server’s current socket is in the time_wait state, the new connection request using the socket will be rejected by the server and a RST packet will be returned.

• SYN packet format error:The SYN packet sent by the client carries other unauthorized tags (such as FIN, URG or other tags), the server will reject the connection and directly return RST
• Firewall policy does not allow: If the client IP is prohibited from connecting, a RST packet will appear in the session. This situation is discussed in detail later.

If a server RST occurs during the session connection establishment phase, it is recommended to analyze the traffic from the server location or troubleshoot the cause of the connection establishment failure at the system level.

In addition to the server, the client may also send a RST packet during the connection establishment phase. As shown in Figures 3 and 4, after the connection establishment has gone through the interaction of SYN packets and SYN/ACK packets, or even after the three-way handshake is completed, the client suddenly sends a RST packet, indicating that the client refuses the connection.

For the situations shown in FIG. 3 and FIG. 4 , it can be directly determined that an abnormality has occurred on the client or the client is launching a port scanning attack.

The traffic in Figure 3 is a typical client-initiated TCP SYN port scan timing diagram, and the traffic in Figure 4 is a typical client-initiated TCP Connect port scan timing diagram. If you encounter such timing diagrams, use policies to block the client IP address and check other traffic of the client IP address to observe whether there are other attack behaviors.

03 How to analyze the RST packet during the data transmission phase?

During the data transmission phase, both the client and the server may send a RST at any time. In this case, the reason for the RST must be that the connection is abnormal.

Common causes of this type of failure are:

• Retransmission limit exceeded: TCP has a retransmission mechanism. When TCP tries to retransmit multiple times but fails to receive confirmation from the other party (or the confirmation sent by the other party cannot be received and processed, such as sequence number error, checksum error and other field errors), the retransmitting party will think that there is an error in the connection, stop retransmitting and send a RST packet to reset the connection . As shown in Figure 5.

• Connection without data exchange for a long time: When there is no data exchange between the client and the server for a long time (for example, 120 seconds), one of the parties may think that the session has timed out and will send an RST packet to reset the connection, as shown in Figure 6. In the scenario where the traffic between the client and the server passes through a load balancer or firewall, the RST packet may also be sent by the load balancer or firewall.

If an RST occurs during the session interaction phase, it is recommended to determine whether the reset is caused by an error or timeout by analyzing the timing diagram. If you suspect that the RST packet is sent from an intermediate device, you can determine the source of the RST packet by comparing the traffic at multiple locations during the same period, thereby further troubleshooting the cause of the fault.

04 What does the timing diagram of a session blocked by a firewall look like?

When the action after the ACL or security policy is matched is Reject, or the security device is deployed in bypass mode and cannot directly discard traffic, the security device will send RST packets to process the session hit by the policy, and the session hit by RST will be interrupted. The session blocked by the security policy is shown in Figure 7.

As can be seen from Figure 7, the server responded to the SYN/ACK packet and immediately replied with a RST. This is because the device that sent the RST packet is an intermediate security device. When bypass blocking is performed, it can only block by sending RST, but cannot intercept the SYN/ACK packet sent by the server.

Therefore, from the client side, it can be seen that the server “replies” to both the SYN/ACK packet and the RST packet at the same time.

If you want to know why we can determine that this RST must be a bypass blocking packet, you can carefully observe these RST packets. The RST packets in Figures 1, 2, 4, 5, and 6 in this article are all “true RST packets” carrying the RST and ACK flags, while the RST packets in Figures 3 and 7 are “fake RST packets” that only carry the RST flag.

In addition, if you can compare the IP TTL of the RST packet and the SYN/ACK packet, you can find that the IP TTL of the two packets may be different (some devices can simulate the IP TTL of the TCP RST packet), indicating that they come from two different network locations, as shown in Figure 8:

Figure 9 describes a bypass blocking that is easier to understand: after the client sends a ClientHello packet, the bypass device in the middle blocks the client but does not send an RST to the server, causing the server to believe that the session is not interrupted and continue to send subsequent ServerHello packets:

In short, if an RST packet appears during a session, you need to consider whether the RST packet is sent by an intermediate bypass blocking device. You can use the RST bit, sequence number, IP TTL, etc. of the RST packet, or directly perform multi-point packet capture and comparison analysis to make a comprehensive judgment.

05 How to analyze the RST packet during the disconnection phase?

During the disconnection phase, RST packets may also appear. This is usually because the party receiving the FIN packet or the intermediate load or security device has an optimization mechanism for disconnection.

Because if the session is terminated after four FIN packets are disconnected, the party that sends the FIN packet first will go through the TCP TIME_WAIT state and will enter the close state after 2MSL time, completely closing the connection.

During the 2MSL period, the socket is unavailable . For a session reset by RST, there is no TIME_WAIT state. Therefore, after a FIN packet appears in some connections, the load/security device thinks that the connection can be terminated and sends a RST packet to quickly close the session.

Although this method is fast, it does not comply with the TCP protocol standard. The session disconnected by RST is shown in Figures 10 and 11:

In the life cycle of a TCP connection, the RST packet plays a key role. By analyzing the timing of the RST packet through the timing diagram, we can see the causes and impacts of the RST packet during the connection establishment, data transmission, abnormal blocking, and connection disconnection stages.