Mastering TCP States: A Comprehensive Guide to Diagnosing Network and System Failures

 

Understanding the various TCP states can be extremely helpful in diagnosing and pinpointing network or system failures.

1. Common TCP Network Commands

Before understanding TCP, first familiarize yourself with a few commands:

1.1. Linux Commands to Check TCP Status:

1) netstat -nat: View the count of TCP states.
2) lsof -i:port Detect open socket conditions.
3) sar -n SOCK: Check the number of TCP connections created.
4) `tcpdump -iany tcp port 9000`: Capture packets for TCP port 9000.
5) `tcpdump dst port 9000 -w dump9000.pcap`: Capture packets for TCP destination port 9000 and save to pcap file for Wireshark analysis.
6) `tcpdump tcp port 9000 -n -X -s 0 -w tcp.cap`: Capture packets for TCP/HTTP target port 9000 and save to pcap file for Wireshark analysis.

1.2. Common Network Testing Commands;

1) ping: Tests the normalcy of network connections, mainly to evaluate latency, jitter, and packet loss rate.

However, many servers disable responses to ping requests to prevent attacks. Therefore, ping is generally used to test connectivity. After issuing the ping command, you will receive feedback information from the target, which includes the target’s IP address and TTL. TTL (Time to Live) is an 8-bit field in the IPv4 header that specifies the maximum number of network segments an IP packet can pass through before being discarded by a router. For instance, if an IP packet is set with a TTL of 64 before being sent from the server, and you receive a TTL of 56 in the feedback using the ping command, it indicates that the packet traversed 8 routers, with the TTL decreasing by 1 for each router it passed.

2) traceroute: A tool used to track the route packets take to reach a network host.

traceroute hostname

3) pathping: This is a route tracing tool that combines the functionalities of the ping and tracert commands with additional information that these two tools do not provide, integrating the capabilities of both.

pathping www.baidu.com

4) MTR: It combines ping, nslookup, and tracert to analyze network characteristics.

5) nslookup: A utility used for domain name resolution, typically employed to check whether the DNS settings on the local machine are configured correctly.

2. TCP Three-Way Handshake Connection States

In the realm of TCP/IP networking, establishing a reliable connection between a client and a server involves a sequence referred to as the “three-way handshake.” This process ensures that both parties are ready to transmit data and establishes parameters for the connection. Below are the states associated with this handshake:

1. SYN-SENT: This state occurs when the client sends a SYN (synchronize) packet to initiate a connection.

2. SYN-RECEIVED: This state occurs when the server receives the SYN packet and responds with a SYN-ACK (synchronize-acknowledge) packet.

3. ESTABLISHED: This state signifies that the connection has been successfully established after the client acknowledges the SYN-ACK packet by sending an ACK (acknowledge) packet back to the server.

Understanding these states is critical for diagnosing and troubleshooting connection issues in network environments.

TCP is a connection-oriented protocol, so before both parties can send data, a connection must first be established.

2.1. Client-Server Three-Way Handshake Process:

When the client side calls the socket function, it is equivalent to the client side generating a socket in the Closed state.

(1) First Handshake SYN:The client side invokes the connect function, and the system randomly assigns a port to the client. Together with the parameters passed into connect (the server’s IP and port), this forms a connection tuple. The client sends a TCP packet with the SYN flag set to the server. This is packet 1 in the three-way handshake process. The connect call puts the client’s socket into aSYN_SENTState, waiting for server confirmation; SYN: Synchronize Sequence Number(Synchronize Sequence Numbers)。

(2) Second handshake SYN+ACK:The server receives the SYN packet and must acknowledge the client’s SYN (ack=j+1), while simultaneously sending its own SYN packet (syn=k), which is a SYN+ACK packet, at this point the server enters theSYN_RECVStatus;

( 3) Third Handshake ACK:Upon the client receiving the SYN+ACK packet from the server, it sends an acknowledgment packet, ACK (ack=k+1), back to the server. Once this packet is transmitted, the client and server enter theESTABLISHEDStatus, the three-way handshake is complete. The connection is now ready for read and write operations.

A complete three-way handshake consists of: Request—Response—Confirmation.

The TCP protocol establishes a connection through three segments, a process known as the three-way handshake, as illustrated in the figure below.

Server-related function interfaces: When the server invokes the socket function call, it generates a listening socket in a Closed state on the server side. The server then calls the bind operation, associating the listening socket with a specified address and port, followed by the listen function. At this point, the system allocates an incomplete queue and a completed queue for it, enabling the listening socket to accept connections from the client. The listening socket is now in the LISTEN state. When the server executes the accept operation, it retrieves a completed client connection from the completed queue. Simultaneously, a session socket is created on the server side for communication with the client-side socket, and this session socket enters the ESTABLISH state.

From the diagram, it is evident that when the client initiates a `connect` call, it triggers a connection request by sending a SYN J packet to the server, causing `connect` to enter a blocking state. The server detects this connection request upon receiving the SYN J packet and calls the `accept` function to process the request, sending back a SYN K and ACK J+1 to the client. At this point, `accept` becomes blocked. Once the client receives the server’s SYN K and ACK J+1, the `connect` call returns, confirming the SYN K. When the server receives the ACK K+1, the `accept` call returns. At this juncture, the three-way handshake is complete and the connection is established.

We can use Wireshark to capture packets, clearly seeing and analyzing the specific process:

First Handshake:It seems like the content provided is not part of a WordPress post text. If it is part of a code snippet or an error message, could you provide more information or context so I can assist you accordingly? If you have text content from a WordPress post that needs translationSecond Handshake:The server receives a SYN packet and must acknowledge the client’s SYN (ACK=j+1=1), while simultaneously sending its own SYN packet (SYN=0).

Third Handshake:The client receives the server’s SYN+ACK packet and sends an acknowledgment packet ACK (ack=k+1) to the server.

2.2. Specific States of Establishing a Connection:

1) LISTENING: Listening for connection requests on TCP ports from remote sources.

First, the server needs to open a socket for listening, and the status is LISTEN.

When a service is provided, it will be in a LISTENING state. The change in TCP states corresponds to the state changes of specific ports; offering a service means opening a port. For example, providing a web service defaults to opening port 80, and providing an FTP service defaults to port 21. When a service is provided but not yet connected, it remains in the LISTENING state. Once the FTP service is started, it first enters a LISTENING state. While in the LISTENING state, the port is open and waiting for connections, but has not been connected yet. It’s like having your house door open, but nobody has come in yet.

The primary concern when observing the LISTENING state is to identify which ports are open on the local machine and determine which programs have opened them. Closing unnecessary ports is a very important aspect of ensuring security. Each service port corresponds to a specific service (application), and stopping that service will close the corresponding port. For example, to close port 21, simply stop the FTP service in IIS. For more on this topic, please refer to other articles.

If you unfortunately fall victim to a trojan on a service port, the trojan will also open a port in the LISTENING state.

2) SYN-SENT: Client SYN_SENT State

After sending a connection request, the client waits for a matching connection request: The client initiates an active open by calling `connect` through the application. Consequently, the client’s TCP sends a SYN to request the establishment of a connection, after which the state transitions to SYN_SENT. /*The socket is actively attempting to establish a connection. Waiting for a matching connection request after sending the connection request.*/

When requesting a connection, the client must first send a synchronization signal to the target machine, at which point the state is SYN_SENT. If the connection is successful, it changes to ESTABLISHED. Normally, the SYN_SENT state is very brief. For example, when accessing the website http://www.baidu.com, you can observe the connection behavior using TCPView if it’s a standard connection.IEXPLORE.EXE (IE) established connections are often found quickly transitioning from SYN_SENT to ESTABLISHED, indicating a successful connection. The SYN_SENT state might be fleeting and not easily observable.

If numerous SYN_SENT states are detected, there are generally a few scenarios to consider: firstly, the website you’re trying to access may not exist or there could be poor network conditions. Secondly, using scanning software to scan a network segment might also result in multiple SYN_SENT states. Lastly, a potential virus infection, such as the “Blaster” worm, might be responsible. When such a virus activates, it scans other machines, leading to many SYN_SENT states appearing.

3) SYN-RECEIVED: Server-side handshake status SYN_RCVD

After receiving and sending a connection request, wait for the other party to confirm the connection request.

When the server receives a synchronization signal from the client, it sets the flag bits ACK and SYN to 1 and sends them to the client. At this point, the server is in the SYN_RCVD state. If the connection is successful, it transitions to ESTABLISHED. Under normal circumstances, the SYN_RCVD state is very short-lived.

If you notice a significant number of SYN_RCVD states, it is possible that your machine is under a SYN Flood DoS (Denial-of-Service) attack.

The principle of a SYN Flood attack is:

During the three-way handshake, attack software sends a SYN connection request to the targeted server (the first step in the handshake), but this address is spoofed. For example, the attack software might randomly forge addresses like 51.133.163.104 or 65.158.99.152. When the server receives the connection request, it sets the ACK and SYN flags to 1 and sends them to the client (the second step in the handshake). However, these client IP addresses are all spoofed, so the server cannot locate the client machine, meaning the third step in the handshake cannot be completed.

In such cases, the server typically retries (sending SYN+ACK to the client again) and, after a certain period, discards this incomplete connection. This duration is referred to as the SYN Timeout, which is generally measured in minutes (approximately 30 seconds to 2 minutes). If a user experiences an anomaly causing a server thread to wait for 1 minute, it is not a major issue. However, if a malicious attacker simulates this scenario on a large scale, the server will consume a significant amount of resources to maintain an exceedingly large half-open connection list—tens of thousands of half connections. Even the simple act of saving and traversing them can consume a substantial amount of resources.CPUThe time and memory, not to mention constantly retrying SYN+ACK on this list of IPs. From the perspective of legitimate clients, the server becomes unresponsive; this situation is what we refer to as:The server-side has experienced a SYN Flood attack (SYN flood attack).It seems like there might have been a mistake or the input

4) ESTABLISHED: Represents an open connection.

The ESTABLISHED state indicates that two machines are currently exchanging data. Monitoring this state primarily involves observing which program is in the ESTABLISHED state.

There are many connections in the ESTABLISHED state on the server: You can detect this by using `netstat -nat | grep 9502` or by using `lsof -i:9502`.

When the client doesn’t proactively close the connection but it gets disconnected: i.e., the FIN sent by the client is lost or wasn’t sent:
If at this point the client had sent a FIN packet when disconnecting, the server will be in the CLOSE_WAIT state.
If the client did not send a FIN packet when disconnecting, the server will still display an ESTABLISHED state.
As a result, the client reconnects to the server.
The newly connected client (i.e., the one that just reconnected) will definitely be in the ESTABLISHED state on the server side. If the client repeatedly exhibits this behavior, the server will accumulate a large number of false ESTABLISHED and CLOSE_WAIT connections.
The ultimate result is that new clients are unable to connect, yet using netstat, one can still see a connection that is established and shows as ESTABLISHED, but it is never able to enter the program code.

3. TCP closes the four-way handshake related status

Since the TCP connection is full-duplex, each direction must be closed independently. The principle here is that once a party has completed its data transmission, it can send a FIN to terminate the connection in that direction. Receiving a FIN only indicates that there is no data flow in that direction; a TCP connection can still send data after receiving a FIN. The party that initiates the closing process will perform an active close, while the other party will perform a passive close.

3.1. The process involves a four-way handshake to close the connection, using TCP states.

Establishing a connection requires a three-way handshake, whereas terminating a connection goes through a four-way handshake. This is due to TCP’s half-close mechanism, as shown in the figure:

Invocation process and corresponding function interface:

• 1) The client sends a FIN: Used to close data transmission from client A to server B,When the client wants to terminate its connection with the server, the client (i.e., an application process) initiates the connection termination by calling close, actively closing the connection. At this point, TCP sends a FIN M; the client side enters theFIN_WAIT1Status.
• 2) The server acknowledges the client’s FIN:When the server receives a FIN M, it executes a passive close. It acknowledges this FIN by sending an ACK back to the client. The acknowledgment number is the received sequence number plus 1 (Segment 5). Like SYN, a FIN will occupy a sequence number. Once the server sends an ACK back to the client, the client enters theFIN_WAIT2Status, the server is inCLOSE_WAITStatus. Its reception is also passed to the application process as an end-of-file marker, because receiving a FIN indicates that the application process will no longer receive additional data on the respective connection;
• 3) Server B sends a FIN to terminate its connection with Client A:After some time, when the server-side detects the client’s closure operation (read returns 0). The server, having received the end-of-file, invokes close to shut down its socket. This causes the server’s TCP to also send a FIN N; at this moment, the server’s state isLAST_ACK。
• 4) Client A sends back an ACK message to confirm:When the client receives a FIN from the server, the client’s socket is in theTIME_WAITstate, it will send an ack confirmation to the server side, and at this point, once the server receives the ack confirmation, this socket is in a CLOSED state.

In this way, there is a FIN and an ACK in each direction.

3.2. Specific States of the Four-Way Handshake for Terminating a Connection

1) FIN-WAIT-1:

Waiting for a remote TCP connection termination request or an acknowledgment of a previous connection termination request.

The application in the active close scenario calls `close`, causing its TCP to send a FIN request to actively close the connection, subsequently entering the `FIN_WAIT1` state. /* The socket is closed, and the connection is shutting down. It waits for the remote TCP’s connection termination request or the acknowledgment of a previously sent connection termination request. */

If the server experiences a shutdown and then restarts, use netstat -nat to observe, and you will see manyFIN-WAIT-1The status. This is because the server currently has many client connections. If you directly shut down the server, it won’t be able to receive the client’s ACK.

2) FIN-WAIT-2: Waiting for a connection termination request from the remote TCP

This is the well-known half-closed state, which occurs after the client and server perform a two-way handshake while closing a connection. In this state, the application can still receive data, but it can no longer send data. However, it is also possible for the client to remain in the FIN_WAIT_2 state while the server remains in the WAIT_CLOSE state, until the application layer decides to close this state.3) CLOSE-WAIT: Awaiting a connection termination request from the local user.

When the TCP of the passive close end receives a FIN, it sends an ACK in response to the FIN request (its reception is also relayed to the upper-layer application as an end-of-file indicator) and enters the CLOSE_WAIT state. /* The remote end has shut down, waiting for the socket to close. Waiting for a connection termination request to be sent from the local user */4) CLOSING: Waiting for acknowledgment of the termination of the connection from the remote TCP.

/* Both sockets are shut down but we still don’t have all our data sent. Await confirmation of connection termination from the remote TCP */

In reality, this situation is quite rare and represents an uncommon exception. Typically, when one side sends a FIN packet, it is expected that the corresponding ACK packet from the other side would be received first (or simultaneously), followed by the FIN packet from the other side. However, the CLOSING state indicates that one side has sent a FIN packet but, instead of receiving the other’s ACK packet, it receives the other side’s FIN packet as well. Under what circumstances might this occur?

There are two scenarios that might lead to this state:
1. If both parties close the connection almost simultaneously, it is possible for both to send FIN packets at the same time.
2. If the ACK packet is lost, and the other side’s FIN packet is sent quickly, the FIN might arrive before the ACK.

4) LAST-ACK: Waiting for the acknowledgment of the original connection termination request sent to the remote TCP.

After a passive close, when an application that has received an end-of-file indicator calls CLOSE to terminate the connection, this causes its TCP to also send a FIN (finish) and wait for the peer’s ACK (acknowledgment). It then enters the LAST-ACK state. /* The remote end has shut down, and the socket is closed. Waiting for acknowledgement. Waiting for confirmation of the connection termination request previously sent to the remote TCP. */

When performing a concurrent stress test, if the stress test client suddenly disconnects, the server will notice manyLAST-ACK。5) TIME-WAIT: Waiting for sufficient time to ensure the remote TCP has received the acknowledgment for the connection termination request.

After the active party receives a FIN, TCP sends an ACK packet and enters the TIME-WAIT state. /* The socket is waiting after close to handlepacketsstill in the network. Wait a sufficient amount of time to ensure the remote TCP receives acknowledgment of the connection termination request */

The TIME_WAIT state, also known as the 2MSL state, occurs after TIME_WAIT2 when the final ACK datagram has been sent. This state is a safeguard against the possibility that the last handshake datagram did not arrive at the other party (note that this is not a fourth handshake step; rather, it’s a fail-safe for the fourth step of the handshake process). This state largely ensures that both parties can terminate the connection cleanly.

Due to the TIME-WAIT state (also known as the 2MSL state, where “2MSL” stands for twice the maximum segment lifetime) of a socket—defined as a combination of an IP address and port—it is impossible for an application to reuse the same socket within this 2MSL duration. This situation is manageable for client applications, but poses issues for server applications, like HTTP servers, which always need to use the same port to provide services. If an HTTP server is started within the 2MSL period, an error will occur indicating that the socket is in use. To circumvent this issue, servers adopt a concept known as the quiet time, which suggests that although the server can be restarted during the 2MSL period, it must silently wait for this time to elapse before initiating the next connection.

For more details, please see:“TIME_WAIT causing Cannot assign requested address error”

6) CLOSED: No connection state

Upon receiving the ACK packet, the passive close side transitions to the closed state, indicating the end of the connection. /* The socket is not being used. No connection status. */

4. TCP State Transition Roadmap

Client/Server Two Routes Elaborate on the TCP State Transition Diagram:

This appears to be a relatively complex state transition diagram because it encompasses two parts: server state transitions and client state transitions. By examining the diagram from a specific perspective, it becomes much clearer. Here, both the server and client are not absolute; the entity sending data is the client, while the entity receiving data is the server.

State Transition Diagram for Client Applications

The client’s status can be represented by the following process:

CLOSED->SYN_SENT->ESTABLISHED->FIN_WAIT_1->FIN_WAIT_2->TIME_WAIT->CLOSED

The process described above is the standard procedure under normal conditions. As illustrated in the book’s diagram, when establishing a connection, once the client receives the ACK for the SYN packet, the client opens a data interaction connection. Typically, the connection is terminated by the client; after the client closes the application, it goes through states such as FIN_WAIT_1 and FIN_WAIT_2. These state transitions correspond to the four-way handshake mentioned earlier for terminating the connection.

State Transition Diagram of Servers

The status of the server can be represented using the following process:

CLOSED->LISTEN->SYN_RECEIVED->ESTABLISHED->CLOSE_WAIT->LAST_ACK->CLOSED

When establishing a connection, the server enters the data exchange state only after the third handshake, while closing the connection happens after the second handshake during the closure process (note that it’s not the fourth). After closing, the server must wait for the client to send the final ACK packet before it can return to the initial state.

Other State Transitions

There are some additional state transitions, which are summarized as follows for both the server and client sides.

LISTEN->SYN_SENT, the explanation for this is quite simple. Sometimes a server needs to open a connection too.

SYN_SENT->Upon receiving a SYN packet, both the server and the client in the SYN_SENT state must send an ACK packet in response, transition their state to SYN_RECEIVED, and prepare to enter the ESTABLISHED state.

SYN_SENT->CLOSED, in the event of a send timeout, it will revert to the CLOSED state.

SYN_RECEIVED -> LISTEN, if an RST packet is received, it will return to the LISTEN state.

SYN_RECEIVED to FIN_WAIT_1, this transition means it can bypass the ESTABLISHED state and directly move to the FIN_WAIT_1 state, waiting for closure.

How can you firmly engrave this image in your mind? You must have a deep understanding of every state in this image and the processes involved in transitioning between them. It’s essential not to remain at a superficial level of understanding. Below is a detailed analysis of the 11 states depicted in this image to enhance memorization! But before that, let’s review the three-way handshake process for establishing a TCP connection and the four-way handshake process for closing it.

5. Specific Issues

1. Why does establishing a connection protocol require a three-way handshake, whereas closing a connection is a four-way handshake?

This is because when a server-side SOCKET is in the LISTEN state and receives a connection request with a SYN packet, it can combine the ACK and SYN into one packet to send. However, when closing a connection, receiving a FIN packet from the other side merely indicates that they no longer have data to send to you; it does not necessarily mean you have sent all your data to them. Therefore, you may not immediately close the SOCKET. In other words, you might still need to send some data to them before transmitting your FIN packet, indicating your agreement to close the connection. As a result, the ACK packet and the FIN packet are usually sent separately in this context.

2. What is 2MSL?

MSL stands for Maximum Segment Lifetime, which refers to the longest time a segment can exist in the network before being discarded. Since a TCP segment is the data part of an IP datagram, please see the article “Naming of Data in Each Layer of the Network” for specific terminology. In the IP header, there is a TTL field, which is short for time to live. This TTL is set to an initial value by the source host, but it records the maximum number of routes an IP datagram can traverse. Each router it passes through decreases this value by 1. When it reaches 0, the datagram is discarded and an ICMP message is sent to the source host. RFC 793 specifies MSL as 2 minutes; however, practical applications often use 30 seconds, 1 minute, and 2 minutes.

2MSL is twice the MSL. The TIME_WAIT state in TCP, also known as the 2MSL wait state, occurs when one side of the TCP connection initiates an active close. After the final ACK packet, the fourth handshake packet is sent, completing the third handshake, and then it enters the TIME_WAIT state. The main reason to wait for 2MSL is to ensure that if the last ACK is not received by the other side, the other side, after a timeout, will retransmit the FIN packet from the third handshake. The actively closing side, upon receiving the retransmitted FIN packet, can send another ACK. During the TIME_WAIT state, neither end can use the ports until the 2MSL period ends. Any late packets during the 2MSL wait stage are discarded. However, in practical applications, setting the SO_REUSEADDR option can reuse this port without waiting for the 2MSL period to end.

This is because, although both sides have agreed to close the connection and all four handshake packets have been coordinated and sent, theoretically returning to the CLOSED state (similar to transitioning from the SYN_SEND state to the ESTABLISH state) could happen:

On one hand, it’s a reliable way to terminate a TCP full-duplex connection. If the final ACK is lost, the passively closing end will retransmit the FIN. Thus, the actively closing end needs to maintain state information to allow it to re-send the final ACK.

On the other hand, because we have to assume the network is unreliable and you cannot guarantee that the last sent ACK will be received by the other side, the socket in the LAST_ACK state may not receive the ACK message in time, prompting it to retransmit the FIN message. The purpose of the TIME_WAIT state is to be able to re-send potentially lost ACK messages.

During the TCP 2MSL wait period, the connection (4-tuple) cannot be reused, and any late packets will be discarded. Imagine if there was no 2MSL restriction, and a new connection happened to satisfy the previous 4-tuple; in this case, delayed packets on the network might interfere with newly established connections.

3. Why does the TIME_WAIT state need to wait for 2MSL before it can return to the CLOSED state?

First,Ensuring a Reliable Termination of a TCP Full-Duplex Connection: To ensure that the final ACK packet sent by the active side A successfully reaches the passive side B, allowing the passive side to enter the CLOSED state.

Referring to the above diagram, in the four-way handshake protocol, when B sends a FIN+ACK to A, A needs to send back an ACK+Seq packet to B. At this point, A enters the TIME_WAIT state. However, this ACK packet might fail to send and get lost, making it impossible for B, which is in the LAST-ACK state, to receive the acknowledgment for its sent FIN+ACK packet. If this happens, B will timeout and retransmit the FIN+ACK packet, and A should expect to receive this retransmitted FIN+ACK packet within the 2MSL time frame. If A does not wait during the TIME_WAIT state but instead releases the connection immediately after sending the ACK packet, it will not be able to receive B’s retransmitted FIN+ACK packet, and consequently, it will not send another acknowledgment packet. This would prevent B from correctly proceeding into the CLOSED state.Second,Prevent obsolete connection request segments from appearing in this connection.Once A has sent the ACK segment, it waits for 2MSL (Maximum Segment Lifetime) time.…all segments generated during the lifetime of this connection can be removed from the network. This ensures that no old connection request segments appear in the next new connection.

Suppose there is a TCP connection between port XXX1 on A and port 80 on B. We close this connection and establish another one at the same IP address and port after some time. The latter connection becomes a reincarnation of the former due to the identical IP addresses and port numbers. TCP must prevent old duplicate packets from a terminated connection from reappearing and being mistakenly interpreted as belonging to a new reincarnation of the same connection. To achieve this, TCP will not initiate a new incarnation while the connection is in the TIME_WAIT state. Since TIME_WAIT lasts twice the Maximum Segment Lifetime (MSL), it is enough to ensure that packets can survive for at most MSL seconds in one direction before being discarded, and similarly, acknowledgments can survive for at most MSL seconds in the opposite direction. By enforcing this rule, we ensure that any duplicate packets from a previous incarnation of a TCP connection have already disappeared from the network by the time a new connection is successfully established.

4. Resolve numerous TIME_WAIT states found on the Linux system.

If your Linux system encounters a significant number of connections in the TIME_WAIT state, you can address this by adjusting kernel parameters. Edit the `/etc/sysctl.conf` file and add the following entries:

bash
net.ipv4.tcp_max_tw_buckets=5000 # Maximum number of TIME-WAIT sockets
# Note: Enabling this setting is not recommended as it may cause connection issues (RST) in NAT mode
net.ipv4.tcp_tw_reuse = 1 # Enables reuse. Allows TIME-WAIT sockets to be reused for new TCP connections. Default is 0 (disabled)
net.ipv4.tcp_tw_recycle = 1 # Enables fast recycling of TIME-WAIT sockets in TCP connections. Default is 0 (disabled)

Then run `/sbin/sysctl -p` to apply the parameters.

6. Explanation of TCP Flags

At the TCP layer, there is a FLAGS field that contains the following flags: SYN, FIN, ACK, PSH, RST, URG. For our daily analysis, the first five flags are mainly useful.

1. Field Meaning:

1. SYN indicates initiating a connection:

The “Synchronize Sequence Numbers” field is valid. This flag is only effective during the TCP connection establishment phase, known as the three-way handshake. It prompts the server side of the TCP connection to acknowledge the sequence number, which is the initial sequence number of the TCP connection initiator (usually the client). Here, the TCP sequence number can be viewed as a 32-bit counter ranging from 0 to 4,294,967,295. Each byte of data exchanged through the TCP connection is associated with a sequence number. The sequence number field in the TCP header includes the sequence number of the first byte in the TCP segment.

2. FIN indicates closing the connection:3. ACK denotes acknowledgment:

Ensure the Acknowledgement Number field is valid. In most cases, this flag is set. The Acknowledgement Number field in the TCP header contains the acknowledgement number (w+1, Figure-1), which is the next expected sequence number. It also indicates that the remote system has successfully received all data.

4. PSH indicates data transmission:

5. RST indicates a connection reset:Reset flag active. Used to reset the corresponding TCP connection.

1. Field Combination Meaning:

In such cases, an ACK can be used together with SYN, FIN, and other flags. For instance, SYN and ACK can both be set to 1 simultaneously, indicating a response after a connection is established. If there is only a single SYN, it merely signifies the initiation of a connection. The process of TCP handshaking is manifested through such ACKs. However, SYN and FIN will not be 1 simultaneously because the former indicates the establishment of a connection, while the latter indicates the termination of a connection. Typically, RST is only set to 1 after FIN, signifying a connection reset. Generally, when a FIN packet or RST packet appears, we consider the client and…ServerThe connection was reset;

Both the RST and ACK flags have been set, and it includes an ACK number. It is very clear that this packet acknowledges previously received packets while simultaneously terminating the TCP connection.

When SYN and SYN+ACK packets appear, we consider that a connection has been established between the client and the server.

A PSH value of 1 typically only appears in packets where the DATA content is not zero, meaning that a PSH value of 1 indicates that actual TCP payload data is being transmitted.

The establishment and closure of a TCP connection are accomplished through a request-response model.

7. Server Handling of Client Unexpected Disconnections in TCP Communication

Reference URL: http://blog.csdn.net/kkkkkxiaofei/article/details/12966407

If a TCP connection is normally closed by the other party, meaning the other party properly called `closesocket(s)` or `shutdown(s)`, then the aforementioned `Recv` or `Send` calls can immediately return with an error. This is because `closesocket(s)` or `shutdown(s)` involves a standard shutdown process, which informs the other side, “The TCP connection has been closed; you don’t need to send or receive messages anymore.”

However, if a disconnection occurs unexpectedly, the client (a 3G mobile device) does not properly close the socket. Both parties do not disconnect the connection following the four-way handshake procedure specified in the protocol.

At this point, the party currently executing the Recv or Send operation will continuously wait due to the lack of any connection interruption notification, essentially becoming stuck for an extended period.

In such cases, if one party has already closed or abnormally terminated the connection while the other party remains unaware, we refer to this type of TCP connection as a half-open connection.

The way to resolve unexpected interruptions is by utilizing a keep-alive mechanism. The keep-alive mechanism can be implemented either at a low level or through self-implementation.

1. Write your own heartbeat packet program.

Simply put, this involves adding a thread to your own program that periodically sends data packets to the other end. It checks for an ACK, and if it exists, the connection is normal; if not, the connection is broken.

2. Enable the keepAlive mechanism in TCP programming.

1. Both Parties Establish Heartbeat (Self-Implementation)

Typically, the client sends heartbeat packets, and the server does not respond to the heartbeat. Instead, it periodically checks to determine if the time interval since the last heartbeat has exceeded a set timeout (a timeout defined by the user). The server does not actively send heartbeats to avoid adding communication load, thereby reducing strain.

However, this will result in three scenarios:

Case 1.

Due to network latency or similar issues, if a client sends a heartbeat much later (without actually being disconnected), the server might interpret this delay as a timeout based on its own settings and proceed to close the socket. If the client has a reconnection mechanism in place, it will attempt to reconnect. To ensure that a properly functioning client is not inadvertently disconnected, it is crucial to select “send” during a ShutDown, indicating the closure of the send channel, while still allowing the server to receive data. This way, if the client happens to be transmitting critical information, it can still be received, right?

Situation 2.

The client has not sent a heartbeat for a long time, indicating that it has disconnected itself. Before it restarts, the server has already determined its timeout and proactively closed the connection, completing the four-step TCP termination process successfully.

Case 3.

The client hasn’t sent a heartbeat for a long time, indicating that it disconnected on its own. Before it restarted, the server’s polling had not yet determined it was timed out, and the client had already reconnected without an active close.

At this point, if the client disconnects by sending a FIN packet, the server will enter the CLOSE_WAIT state.

At this point, if the client disconnects without sending a FIN packet, the server side will still display the ESTABLISHED state;

And the newly connected client (which previously got disconnected and reconnected) will definitely be in the ESTABLISHED state on the server side; at this point, there’s an issue where, if through polling, the previous old connection hasn’t yet been detected as timed out (this is normal, as the timer will always have an interval), and during this time, the client repeatedly replays scenario 3, the server will end up with a large number of false ESTABLISHED connections and CLOSE_WAIT connections.

The ultimate result is that new clients are unable to connect, yet using `netstat`, I can still see a connection established with an ESTABLISHED state, but it fails to enter the program’s code. Initially, I suspected that this issue was due to fake ESTABLISHED connections and CLOSE_WAIT connections consuming significant system resources, preventing the program from creating new connections (since every time I encountered this issue, only about 10 clients were connected, but there were more than 40 invalid connections). However, recent tests showed an instance where only 2 or 3 devices were connected within the program, yet there were approximately 8 phantom connections, and new clients couldn’t connect. This led me to realize that my initial assumption was incorrect; it’s unlikely that such few connections would consume a vast amount of resources. If it were dozens, it might be possible. However, it’s certain that this issue arises from devices constantly rebooting, while the server employs simple polling, which cannot handle the issue promptly. As of now, this remains unresolved.

Section 2: Using KeepAlive

In fact, the principle of keepalive is essentially a heartbeat packet embedded in TCP,

Taking the server side as an example, if the current server detects no data transmission for a certain period (default is 7,200,000 milliseconds, which is 2 hours), it will send a keep-alive packet to the client (this keep-alive packet is a combination of an ACK and the current TCP sequence number minus one). At this point, the client should be in one of the following three situations:

1. The client-side is still present, and the network connection is stable. At this point, the client-side will return an ACK. After the server-side receives the ACK, it resets the timer (reinitializes the keep-alive timer) and sends a probe again after 2 hours. If there is data transmission within the 2 hours, the next probe is delayed by another 2 hours based on that time.

2. Client unexpectedly closes, or the network disconnects. In both these scenarios, the client will not respond. The server does not receive a response to its sent probes, and after a certain period (system default is 1000 ms), it will resend a keep-alive packet and will repeat this process a certain number of times (2000 XP 2003 systems default to 5 times).Vistathe system defaults to 10 times).

3. The client previously crashed but has since restarted. In this scenario, the server will receive a response to its keep-alive probe, but this response will be a reset, prompting the server to terminate the connection.

For applications, two hours of idle time is too long. Therefore, we need to manually enable the Keepalive feature and configure reasonable Keepalive parameters.

To modify the global settings, you can edit `/etc/sysctl.conf` and add:

net.ipv4.tcp_keepalive_intvl = 20
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_time = 60

Set the following in the program:

 #include < socket.h="">
  #include < in.h="">
  #include < inet.h="">
  #include < types.h="">
  #include < tcp.h="">cpp
int keepAlive = 1; // Enable the keepalive attribute
int keepIdle = 60; // If there is no data exchange over the connection within 60 seconds, initiate probing
int keepInterval = 5; // The time interval for sending packets during probing is 5 seconds
int keepCount = 3; // Number of probe attempts. If a response is received at the first probe, the remaining two will not be sent.

setsockopt(rs, SOL_SOCKET, SO_KEEPALIVE, (void *)&keepAlive, sizeof(keepAlive));
setsockopt(rs, SOL_TCP, TCP_KEEPIDLE, (void*)&keepIdle, sizeof(keepIdle));
setsockopt(rs, SOL_TCP, TCP_KEEPINTVL, (void *)&keepInterval, sizeof(keepInterval));
setsockopt(rs, SOL_TCP, TCP_KEEPCNT, (void *)&keepCount, sizeof(keepCount));
</></></></></>

In the program, when TCP detects that the peer socket is no longer available (unable to send a probe packet, or the probe packet does not receive an ACK response packet), select will return the socket as readable, and recv will return -1 while setting errno to ETIMEDOUT.

8. List of Linux Error Messages (errno)

Frequent Errors:

22: Parameter error, such as an invalid IP address or missing target port, etc.

101: Network unreachable, such as being unable to ping.

111: Connection refused, such as the target closing the connection, etc.

115: When a connection is set to non-blocking and the target doesn’t respond in a timely manner, this error is returned, and the socket can continue to be used. For example, a socket connection.

Appendix: Linux Errno Table

_ 124 EMEDIUMTYPE_ Wrong medium type
_ 123 ENOMEDIUM__ No medium found
_ 122 EDQUOT___ Disk quota exceeded
_ 121 EREMOTEIO__ Remote I/O error
_ 120 EISNAM___ Is a named type file
_ 119 ENAVAIL___ No XENIX semaphores available
_ 118 ENOTNAM___ Not a XENIX named type file
_ 117 EUCLEAN___ Structure needs cleaning
_ 116 ESTALE___ Stale NFS file handle
_ 115 EINPROGRESS +Operation now in progress

Operation in progress. A blocking operation is ongoing.
_ 114 EALREADY__ Operation already in progress
_ 113 EHOSTUNREACH No route to host
_ 112 EHOSTDOWN__ Host is down
_ 111 ECONNREFUSED Connection refused

1. Connection Refused. Generally occurs when establishing a connection.

Unplug the server’s network cable for testing. When the client sets keep alive, recv returns 0 quickly. The error code ECONNREFUSED (Connection refused) is received first, followed by ETIMEOUT.

2、an error returned from connect(), so it can only occur in a client (if a client is defined as the party that initiates the connection
_ 110 ETIMEDOUT_ +Connection timed out
_ 109 ETOOMANYREFS Too many references: cannot splice
_ 108 ESHUTDOWN__ Cannot send after transport endpoint shutdown
_ 107 ENOTCONN__ Transport endpoint is not connected

Performing read or write operations on a socket that has not established a connection will return this error. The cause of the error is that the socket does not have an identified address. Setsock may also encounter errors.

Another scenario occurs when you receive an RST packet from the other party, indicating that the system has confirmed the connection has been terminated.
_ 106 EISCONN___ Transport endpoint is already connected

Typically, this error occurs when the socket client is already connected, but invoking connect triggers this issue.
_ 105 ENOBUFS___ No buffer space available
_ 104 ECONNRESET_ Connection reset by peer

Connection closed by the remote host. There are several reasons for this: the remote host has stopped the service or restarted; when certain operations encounter a failure, the connection is closed due to the “keep alive” option being set, often appearing alongside ENETRESET.

In a client-server application, if the client exits unexpectedly without releasing or closing the relevant resources, the server will first receive an ECONNRESET error and then an EPIPE error.

The connection was closed by the remote host. There are several possible reasons for this: the remote host has stopped the service and restarted; a failure was encountered while performing certain operations, and because the “keep alive” option is set, the connection was closed, typically occurring alongside ENETRESET.

3. The remote end performed a “hard” or “abortive” shutdown. The application should close the socket because it is no longer available. When this is executed on a UDP socket, this error indicates that a previous send operation returned an ICMP “port unreachable” message.

4. If the client closes the connection, the server’s select does not report an error (does not return -1; using select for non-blocking detection on a single socket), but writing to that socket will result in an error when using send. The error number is: ECONNRESET. Reading (recv) from the socket does not return an error.

5. The error is described as “connection reset by peer,” which means “the other party reset the connection.” This situation typically occurs when the server process terminates before the client process.

The process of proactively closing the call is as follows:

Server-Side Initiated Closure:

1) When the server’s service process is prematurely terminated for some reason, it sends a FIN segment to the client’s TCP. The server side is in a FIN_WAIT1 Status.

2) After the client’s TCP responds with an ACK, the server’s TCP will transition into FIN_WAIT2 Status.

At this point, if the client process has not handled the FIN (for example, it is blocked on other calls and has not closed the Socket), then the client TCP will be in a CLOSE_WAIT
Status.

4) When the client process sends data again to the service TCP in the FIN_WAIT2 state, the service TCP will immediately respond with an RST.

In general, this situation can also trigger other application exceptions. After sending data, the client process often waits to receive data from network I/O, typically through calls like `read` or `readline`. Due to execution timing reasons, if this call occurs before receiving the RST segment, the result is that the client process will encounter an unexpected EOF error. At this point, it typically outputs the “server terminated prematurely” error.

_ 103 ECONNABORTED Software caused connection abort

1. Connection canceled by software. An established connection has been canceled by the host software, possibly due to data transmission timeout or protocol error.

2. This error is described as “software caused connection abort.” The reason is that after the server and client processes complete the “three-way handshake” for a TCP connection, the client TCP sends a RST (reset) segment. From the perspective of the server process, the RST arrives just when the connection has been queued by TCP, waiting for the server process to call accept. POSIX stipulates that the errno value at this point must be ECONNABORTED. The implementation originating from Berkeley handles the aborted connection entirely within the kernel, so the server process will never know that the abort occurred. Generally, the server process can ignore this error and directly call accept again.

When the TCP protocol receives an RST segment, it indicates that there is some kind of error in the connection. The `read` function will return with an error, with the error type being `ECONNRESET`. Furthermore, all subsequent read operations on this socket will return an error. Upon an error return, the return value is less than 0.
_ 102 ENETRESET__ Network dropped connection on reset

The connection is lost when the network resets.

An error was detected due to the “keep-alive” option being set, and the connection was interrupted. Attempting to use the setsockopt operation on an already failed connection will also return this error.
_ 101 ENETUNREACH_ Network is unreachable

Network unreachable. The socket attempted to operate on an unreachable network. This implies that the local software is aware there is no route to the remote host.

– 100 ENETDOWN__ Network is down
– 99 EADDRNOTAVAIL Cannot assign requested address
– 98 EADDRINUSE_ Address already in use
– 97 EAFNOSUPPORT Address family not supported by protocol
– 96 EPFNOSUPPORT Protocol family not supported
– 95 EOPNOTSUPP_ Operation not supported
– 94 ESOCKTNOSUPPORT Socket type not supported

The socket type is not supported. The specified socket type is not supported within its address family. For example, the option SOCK_RAW may be selected, but the implementation does not support SOCK_RAW sockets.
_ 93 EPROTONOSUPPORT Protocol not supported

Unsupported protocol. The identified protocol is either not installed in the system or not implemented. For instance, a function requires a SOCK_DGRAM socket but designates a stream protocol.
_ 92 ENOPROTOOPT_ Protocol not available

The error is not related to a Socket connection. The errno value might indicate that when using the getsockopt system call to obtain the current option status of a socket, this error is triggered if an unsupported option parameter is detected by the system.
_ 91 EPROTOTYPE_ Protocol wrong type for socket

Protocol type error. The identified protocol’s socket function is performing operations on a non-supported socket, such as ARPA Internet.

The UDP protocol cannot be identified as a SOCK_STREAM socket type.
_ 90 EMSGSIZE__ +Message too long

The message body is too long.

A packet sent to a socket is larger than the internal message buffer size or exceeds other network limitations, or the buffer used to receive the packet is smaller than the packet itself.
_ 89 EDESTADDRREQ Destination address required

Please provide the destination address.

Performing operations on a socket requires providing an address. For example, performing a sendto operation on an ADDR_ANY address will return this error.
_ 88 ENOTSOCK__ Socket operation on non-socket

Performing socket operations on a non-socket.
_ 87 EUSERS___ Too many users
_ 86 ESTRPIPE__ Streams pipe error
_ 85 ERESTART__ Interrupted system call should be restarted
_ 84 EILSEQ___ Invalid or incomplete multibyte or wide character
_ 83 ELIBEXEC__ Cannot exec a shared library directly
_ 82 ELIBMAX___ Attempting to link in too many shared libraries
_ 81 ELIBSCN___ .lib section in a.out corrupted
_ 80 ELIBBAD___ Accessing a corrupted shared library
_ 79 ELIBACC___ Cannot access a needed shared library
_ 78 EREMCHG___ Remote address changed
_ 77 EBADFD___ File descriptor in bad state
_ 76 ENOTUNIQ__ Name not unique on network
_ 75 EOVERFLOW__ Value too large for defined data type
_ 74 EBADMSG__ Bad message
_ 73 EDOTDOT___ RFS specific error
_ 72 EMULTIHOP__ Multihop attempted
_ 71 EPROTO___ Protocol error
_ 70 ECOMM____ Communication error on send
_ 69 ESRMNT___ Srmount error
_ 68 EADV____ Advertise error
_ 67 ENOLINK___ Link has been severed
_ 66 EREMOTE___ Object is remote
_ 65 ENOPKG___ Package not installed
_ 64 ENONET___ Machine is not on the network
_ 63 ENOSR____ Out of streams resources
_ 62 ETIME____ Timer expired
_ 61 ENODATA___ No data available
_ 60 ENOSTR___ Device not a stream
_ 59 EBFONT___ Bad font file format
_ 57 EBADSLT___ Invalid slot
_ 56 EBADRQC___ Invalid request code
_ 55 ENOANO___ No anode
_ 54 EXFULL___ Exchange full
_ 53 EBADR____ Invalid request descriptor
_ 52 EBADE____ Invalid exchange
_ 51 EL2HLT___ Level 2 halted
_ 50 ENOCSI___ No CSI structure available
_ 49 EUNATCH___ Protocol driver not attached
_ 48 ELNRNG___ Link number out of range
_ 47 EL3RST___ Level 3 reset
_ 46 EL3HLT___ Level 3 halted
_ 45 EL2NSYNC__ Level 2 not synchronized
_ 44 ECHRNG___ Channel number out of range
_ 43 EIDRM____ Identifier removed
_ 42 ENOMSG___ No message of desired type
_ 40 ELOOP____ Too many levels of symbolic links
_ 39 ENOTEMPTY_ Directory not empty
_ 38 ENOSYS___ Function not implemented
_ 37 ENOLCK___ No locks available
_ 36 ENAMETOOLONG File name too long
_ 35 EDEADLK__ Resource deadlock avoided
_ 34 ERANGE___ Numerical result out of range
_ 33 EDOM____ Numerical argument out of domain
_ 32 EPIPE___ Broken pipe

The receiving end is closed (with no excess data in the buffer), but the sending end is still writing:

1. When a socket is closed, but its socket descriptor is not set to -1, continuing to perform send and recv operations on this socket will result in this error. This error can trigger the SIGPIPE signal, which can terminate the process that generated this EPIPE error. Therefore, it is common practice in network programming to first block this signal to prevent the untimely termination of the process due to the socket not being correctly set.

2、write(..) on a socket that has been closed at the other end will cause a SIGPIPE.

3. The error is described as a “broken pipe.” This typically occurs when a client process ignores (or fails to promptly handle) a socket error and continues to write more data to the service TCP. The kernel then sends a SIGPIPE signal to the client process, which by default will terminate the process (without generating a core dump for this foreground process). From the preceding ECONNRESET error, it can be inferred that writing data to a FIN_WAIT2 state service TCP (which has already acknowledged the FIN segment) is acceptable, but writing to a socket that has received an RST is an error.

– 31 EMLINK___ +Too many links
– 30 EROFS___ +Read-only file system
– 29 ESPIPE___ +Illegal seek
– 28 ENOSPC___ +No space left on device
– 27 EFBIG___ +File too large
– 26 ETXTBSY___ Text file busy
– 25 ENOTTY___ +Inappropriate ioctl for device
– 24 EMFILE___ +Too many open files

Opened too many sockets. For a process or thread, each implementation method has a maximum number of sockets it can handle, which may be either global or localized.

_ 23 ENFILE___ +Too many open files in system
_ 22 EINVAL___ +Invalid argument

Invalid parameter. The provided parameter is illegal. Sometimes it is also related to the current state of the socket, such as when a socket is not in a listening state, calling accept will generate an EINVAL error.
_ 21 EISDIR___ +Is a directory
_ 20 ENOTDIR__ +Not a directory
_ 19 ENODEV___ +No such device
_ 18 EXDEV___ +Invalid cross-device link
_ 17 EEXIST___ +File exists
_ 16 EBUSY___ +Device or resource busy
_ 15 ENOTBLK___ Block device required
_ 14 EFAULT___ +Bad address
_ 13 EACCES___ +Permission denied
_ 12 ENOMEM___ +Cannot allocate memory
_ 11 EAGAIN___ +Resource temporarily unavailable

When reading data, a common situation is encountering an empty underlying buffer. The typical approach is to perform a read operation in a loop, and in asynchronous mode, wait for the read event to occur before proceeding.

1. If the `send` function return value is less than the number of bytes you want to send, it will return `EAGAIN` or `EINTR`.

2. When the return value of `recv` is less than the requested length, it indicates that the buffer no longer has readable data. However, reading again might not necessarily trigger EAGAIN; it might return 0, indicating that the TCP connection has been closed.

3. When the socket is non-blocking and this error is returned, it indicates that the write buffer queue is full. In this case, you can implement a delay before retrying.

4. When receiving data with a non-blocking socket in Linux, you might frequently encounter the message “Resource temporarily unavailable,” corresponding to error number 11 (EAGAIN). This indicates that a blocking operation was attempted in non-blocking mode, and since the operation wasn’t completed, the error is returned. This error does not disrupt socket synchronization and can be ignored; simply continue with recv in the next loop. For non-blocking sockets, EAGAIN is not considered an error.

– 10 ECHILD +No child processes
– 9 EBADF +Bad file descriptor
– 8 ENOEXEC +Exec format error
– 7 E2BIG +Argument list too long
– 6 ENXIO +No such device or address
– 5 EIO +Input/output error
– 4 EINTR +Interrupted system call

A blocking operation was interrupted by a blocking call. If send or receive timeouts are set, you might encounter this error.

Only applicable to blocking mode sockets. When reading or writing blocking sockets, a return value of -1 indicates an error number of EINTR. Additionally, if EINTR occurs, meaning errno is 4 with the error description “Interrupted system call,” the operation should also continue. If the return value of recv is 0, it indicates that the connection has been closed, and the receive operation should end as well.
__ 3 ESRCH___ +No such process
__ 2 ENOENT___ +No such file or directory
__ 1 EPERM___ +Operation not permitted