3-Way Handshake: Start of Relation

It's Just Like a Phone Call

When I first learned about TCP, the term "Connection-oriented protocol" didn't make much sense to me. Then one day, while making a phone call, it clicked. Oh, this is exactly like making a phone call!

Think about when we make a phone call:

1. I dial and say "Hello?" (SYN)
2. The other person picks up. "Hi, can you hear me?" (SYN-ACK)
3. I confirm. "Yes I can. So..." (ACK)

Data transmission starts only after this 3-step greeting. Without this process? I'd be talking to myself, not knowing if the other person is even listening or if the line is dead. When I was debugging network errors in my service, understanding this concept made it so much easier to pinpoint where things went wrong.

It's Not Just a Simple Greeting

At first, I thought "Oh, it's just 3 greetings." But I learned there's much more happening under the hood. It's actually exchanging Sequence Numbers to prepare for reliable communication.

Step 1: SYN (Synchronize)

The client tells the server "Let's connect. My Initial Sequence Number (ISN) is 100." The packet flags are SYN=1, ACK=0, and the client enters SYN_SENT state.

The important part here is that the client sends a randomly generated ISN of 100. It's saying "I'll start numbering my data from this number." I initially wondered "Why not start from 0?" Later I learned it's for security reasons (preventing sequence number prediction attacks) and to distinguish old packets from new ones.

Step 2: SYN-ACK (Synchronize-Acknowledgment)

The server responds to the client: "Got it (ACK 101). I hear you. I'm also connecting. My ISN is 2000 (SYN)." Packet flags are SYN=1, ACK=1, and the server enters SYN_RCVD state.

What's interesting about this step is that it does two things at once:

• ACK 101: Acknowledging the client's 100, asking for 101 next
• SYN 2000: The server also generates its own ISN (2000) and sends it

Since it's bidirectional communication, the server is also saying "I have things to say too." Understanding this made it clear why it's called "3-Way."

Step 3: ACK (Acknowledgment)

The client sends the final confirmation to the server: "Got your 2000 (ACK 2001). Connection established." Packet flags are SYN=0, ACK=1, and the client enters ESTABLISHED state. When the server receives this, it also becomes ESTABLISHED.

From this point on, a logical connection tunnel (Socket) is considered created. Now we can exchange data.

Why 3? Why Not 2?

This was my biggest question when I first learned this. "Why can't we just send SYN, get SYN-ACK, and start?" This is actually a commonly asked question too.

I Experienced the Zombie Packet Problem

Networks are unreliable. Packets can arrive late or out of order. Imagine this scenario:

1. Client sends SYN_A, but it gets delayed due to network congestion
2. Client thinks "No response?" and sends a retry (SYN_B), completes communication, and closes
3. 10 minutes later, the lost SYN_A finally arrives at the server

If it was 2-Way Handshake? The server sees SYN_A and immediately thinks "Oh, new connection!" and establishes the connection, allocating memory. But the client is already done. The server is left talking to empty air, wasting resources.

The 3-Way Solution

With 3-Way Handshake:

1. Server receives the late SYN_A and sends SYN-ACK
2. Client thinks "Huh? I didn't request a connection now" and sends a RST (Reset) packet
3. Server sees RST and thinks "Oh, this is old" and cancels the connection

In other words, the 3rd ACK is the final confirmation stamp saying "this connection request is valid right now." Understanding this made me appreciate why TCP was designed this way.

Breaking Up Requires More Care: 4-Way Handshake

Closing a connection takes 4 steps, not 3. Why? Because there might still be data in transit.

1. Client (FIN): "I'm done sending. Let's close." → FIN_WAIT_1
2. Server (ACK): "Got it. Wait a moment (I have stuff to finish)." → CLOSE_WAIT
3. Server (FIN): "Okay, I'm also done. Let's really close." → LAST_ACK
4. Client (ACK): "Okay. Bye." → TIME_WAIT → Close

The Mystery of TIME_WAIT

The client doesn't die immediately after sending the final ACK. It stays alive in TIME_WAIT state for about 2 minutes. At first I thought "Why not close immediately?" but there's a reason.

If the final ACK gets lost? The server thinks "Didn't they get my FIN?" and retransmits FIN. If the client is already dead? The server can never close properly and gets an error. So the client waits a bit, thinking "Just in case they didn't hear my last message."

I was shocked when I ran netstat -an | grep TIME_WAIT on my service and saw thousands of them. Turns out it was a signal that the client was connecting and disconnecting too frequently. To solve this, I needed to use a Connection Pool. Instead of doing 3-Way Handshake every time (CPU + Latency cost), you pre-establish multiple connections and reuse them.

I Had to See It with Wireshark to Believe It

I believe in seeing things with my own eyes, so I opened Wireshark and captured actual packets.

Normal Connection

1. 192.168.0.2 -> 8.8.8.8 [SYN] Seq=0
2. 8.8.8.8 -> 192.168.0.2 [SYN, ACK] Seq=0 Ack=1
3. 192.168.0.2 -> 8.8.8.8 [ACK] Seq=1 Ack=1

The Seq=0 here is Wireshark's convenience conversion to Relative Sequence Numbers. In reality, it's a random large number like 39281912.

When Server is Down (Connection Refused)

If the server port is closed, the server immediately sends a RST (Reset) packet upon receiving SYN.

1. Client -> Server [SYN]
2. Server -> Client [RST, ACK]

In Java, this throws java.net.ConnectException: Connection refused. I've seen this error countless times while developing, and now I immediately know "Oh, the server process isn't running."

When Firewall is Blocking (Timeout)

If a firewall in front of the server (like AWS Security Group) drops the packet, the server doesn't respond (silent).

1. Client -> Server [SYN]
2. (no response...)
3. Client -> Server [SYN] (Retransmission)
4. (no response...)

The client retries a few times and eventually throws SocketTimeoutException. If the error message is "Refused," it means "server is on but program isn't running." If it's "Timeout," it's 99% likely a "network/firewall issue." Knowing this difference made troubleshooting much faster.

I Came to Understand the TCP State Machine

As a developer, all we can control in code is connect() and close(), but inside the OS kernel, complex state transitions are happening.

LISTEN → ESTABLISHED

1. LISTEN: Server opens a port and waits (ServerSocket)
2. SYN_SENT: Client calls connect()
3. SYN_RCVD: Server receives SYN and sends SYN-ACK (half-connected)
4. ESTABLISHED: Ready for data transmission

If it gets stuck in step 3 (SYN_RCVD), that's a SYN Flooding attack. Hackers send countless SYNs without sending ACKs, filling up the server's backlog queue in a DDoS attack.

ESTABLISHED → CLOSED

1. FIN_WAIT_1: "I want to close" sent
2. FIN_WAIT_2: Received ACK from other side. Waiting for their FIN
3. CLOSE_WAIT: (Other side's perspective) "They're closing. Let me finish up too"
4. TIME_WAIT: After all closing procedures, waiting for stray packets

If it stays in CLOSE_WAIT state? That's a Code Leak. You didn't call input.close(). When I first built my service, I made this mistake and watched the server slow down and eventually die.

Congestion Control: Start Carefully

After the handshake, do we immediately pour out data? No. We need to check the receiver and network conditions.

1. Slow Start: Start with 1 packet, if successful try 2, then 4... exponentially increasing. "Testing the waters"
2. Congestion Avoidance: Once it's somewhat full, increase linearly (1 at a time) carefully
3. Fast Retransmit: If we get 3 out-of-order ACKs, "Oh, it was lost" and immediately retransmit

All of this happens automatically inside TCP (kernel). When we call socket.send(data), the OS performs all these complex algorithms and chunks the data. UDP doesn't have this, so developers have to implement it themselves.

TCP Fast Open: Skipping the Greeting

3-Way Handshake is safe but slow. It wastes 1 RTT (Round Trip Time) every time. Google proposed TCP Fast Open (RFC 7413).

The idea is simple. "We talked yesterday, can we skip the greeting?" On the first connection, you get a Cookie. From the second connection onwards, you send Data + Cookie inside the SYN packet. The server verifies the cookie and processes data even before the handshake completes. It's widely used in modern browsers and Linux servers.

The HOL Blocking Problem

TCP strictly maintains data order (1, 2, 3). If packet 2 is lost? Even if 3, 4, 5 have already arrived, the OS won't give them to the application and waits tightly until packet 2 is retransmitted. The whole stream freezes.

This is why HTTP/3 (QUIC) abandoned TCP and chose UDP. UDP doesn't have order obsession, so streams can flow independently. At first I wondered "Why use UDP?" but understanding this problem made it clear.

Linux Kernel Tuning: Reading the Parameters

Studying Linux kernel parameters (sysctl.conf), I found that environments handling large traffic often need to go beyond the default TCP settings. These are the settings worth knowing about.

Increasing SYN Backlog

When SYN packets flood in, the kernel's backlog queue fills up and even legitimate connections get refused.

# Check SYN backlog size
sysctl net.ipv4.tcp_max_syn_backlog
# 128 (default) -> need to increase to 4096 etc.

Enable SYN Cookies (Security)

Enable cookie technology to defend against SYN Flooding attacks.

sysctl -w net.ipv4.tcp_syncookies=1

Window Scaling (Performance)

The TCP header's Window Size field is limited to 16 bits (65KB). That was enough in the old days, but in the gigabit era, it's woefully insufficient. Enabling Window Scaling allows window sizes up to 1GB through shift operations.

sysctl -w net.ipv4.tcp_window_scaling=1

These settings are typically tuned when operating high-performance middleware like Redis or Kafka.

Graceful Shutdown: Half-Close

Sending FIN in 4-Way Handshake means "I have no more data to send," not "I won't listen." Using this subtle difference is Half-Close.

• socket.close(): Closes both read/write streams. If the other side sends more data, it errors
• socket.shutdown(SHUT_WR): Closes only "write" (FIN sent). But you can still read data from the other side

Think about a client uploading a file to a server. The client finishes sending the file and calls shutdown(WR) to send FIN. But the server might not have sent the "file received successfully" response yet. The client should close only writing and maintain the reading state (reading open) to wait for the server's response. This is the proper way to do graceful shutdown.

Wrapping Up

When I first learned TCP 3-Way Handshake, I thought "It's just 3 greetings." But the deeper I dug, the more I realized how precisely designed this protocol is.

• SYN: "Hello?" (Start conversation)
• SYN-ACK: "I hear you, can you hear me?" (Bidirectional confirmation)
• ACK: "Yes I can" (Final confirmation)

These 3 steps allow us to have reliable communication even on unreliable networks. They prevent zombie packet problems, guarantee bidirectional communication, and exchange sequence numbers to maintain data order.

When debugging network errors in my service, properly understanding this concept allowed me to find problems much faster. The difference between "Refused" and "Timeout," the meaning of TIME_WAIT, the need for Connection Pools... all of these naturally connected once I understood 3-Way Handshake.

In the end, TCP is a "polite protocol." It listens before speaking, and considers the other party before disconnecting. Understanding this philosophy makes network programming much easier.