TCP Tuning for HFT

Here is a detailed, diagram‑by‑diagram explanation of the five TCP‑related topics: Congestion Control, Delayed Acknowledgment, Nagle Algorithm, * Zero‑Copy*, and Protocol Stack Behavior Analysis.

---

1. TCP Congestion Control

Congestion control prevents a sender from overwhelming the network. The most common implementation uses Additive Increase Multiplicative Decrease (AIMD) along with slow start, congestion avoidance, fast retransmit, and fast recovery.

• Slow Start – The congestion window (cwnd) starts at 1 segment and doubles every RTT (exponential growth) until a threshold (ssthresh) is reached.

• Congestion Avoidance – After ssthresh, cwnd increases by 1 segment per RTT (linear growth).

• Packet Loss – Detected by timeout or three duplicate ACKs → ssthresh is set to half of the current cwnd, and cwnd is reduced accordingly ( multiplicative decrease). Fast retransmit/recovery avoid going back to slow start.

Congestion Window vs Time
cwnd
  ^
  |   Slow Start     Congestion Avoidance
  |      /‾‾‾\          /‾‾‾‾‾‾‾‾‾\
  |     /     \        /             \
  |    /       \      /               \
  |   /         \____/                 \______
  |  /         (loss)                         \
  | /                                           \_
  +--------------------------------------------------> Time
     <--ssthresh-->

• Blue = slow start (exponential)

• Red = congestion avoidance (linear)

• Drop = loss event → window halved

---

2. Delayed Acknowledgment

To reduce the number of pure ACK packets, TCP receivers delay sending an ACK. The ACK is typically held for up to 200‑500 ms or until:

• a second full‑sized segment arrives (ACK every other packet), or

• the application sends data (piggybacking).

This improves efficiency, especially on networks with many small messages.

Sender                          Receiver
  |                               |
  | -------- [Data] ------------> |
  |                               |  (Delayed ACK timer starts)
  | -------- [Data] ------------> |
  |                               |  (ACK now sent, often with 2nd data)
  | <-------- [ACK] ------------- |
  |                               |
  | -------- [Data] ------------> |
  |                               |  (Timer expires or data to send)
  | <-------- [ACK+Data] -------- |  (Piggybacked acknowledgment)

Key: Delayed ACK is a receiver‑side behavior, while Nagle is a sender‑side behavior.

---

3. Nagle Algorithm

Nagle’s algorithm reduces the number of tiny packets sent over a TCP connection. It works by:

• Buffering small data chunks sent by the application.

• Sending the buffered data only when either:

  • an ACK is received for previously sent data, or

  • enough data has been buffered to send a full‑sized segment (MSS).

This avoids the “silly window syndrome” but can increase latency for interactive applications.

Application writes          TCP sender
     "H"                    +--------+
     ---->                  |        |   No ACK pending? 
                            |  Nagle |   No → buffer "H"
     "e"                    |  logic |   +------+
     ---->                  |        |   | "He" |
     "l"                    +--------+   +------+
     ---->                      |          |
     "l"                        |          |
     ---->                      |          |
     "o"                        |          |
     ---->                      +----------+
                                | (ACK arrives)
                                v
                         Send buffered "Hello"
                                |

Result: Multiple small writes are combined into one TCP segment.

---

4. Zero‑Copy

Zero‑copy techniques eliminate redundant copying of data between kernel space and user space, reducing CPU overhead and memory bandwidth. Traditional data transfer (e.g., from disk to network) involves multiple copies:

Traditional I/O path:

Disk  --->  Kernel buffer  --->  User buffer  --->  Kernel socket buffer  --->  NIC
            (read)                (copy)            (send)                   (DMA)

Zero‑Copy (e.g., sendfile):

Disk  --->  Kernel buffer  --->  NIC
            (DMA)                 (DMA, descriptor passing)

No copying through user space. Data descriptors are passed directly between kernel subsystems.

Traditional I/O:
App         Kernel
+-----+     +-----+
| buf | <-- | read| <--- Disk
+-----+     +-----+
   |           |
   +--copy--->| send| ---> NIC
              +-----+

Zero-Copy (sendfile):
App         Kernel
+-----+     +-----+          Disk
|     |     |     | <--- DMA
+-----+     +-----+
            |     | ---> NIC (DMA)
            +-----+

Examples: sendfile(), splice(), mmap(), and RDMA.

---

5. Protocol Stack Behavior Analysis

Analyzing the TCP/IP stack behavior involves observing packet exchanges, state transitions, and performance metrics. This is done with tools like Wireshark, tcpdump, ss, or custom kernel instrumentation.

Key aspects analysed:

• Three‑way handshake (SYN, SYN‑ACK, ACK)

• Sequence/acknowledgment numbers (data delivery, retransmissions)

• Window scaling and congestion window evolution

• Retransmission timeouts (RTO) and fast retransmits

• Delayed ACK and Nagle interaction

• Application‑layer protocol timing

Diagram: Packet flow through the stack with capture points:

Application            (e.g., HTTP)
      |
      v
   TCP Layer           <-- analyse: cwnd, rtt, state (ESTAB, FIN_WAIT...)
      |                    (tcpdump, ss, tcptrace)
      v
   IP Layer            <-- analyse: fragmentation, TTL, options
      |
      v
   Link Layer          <-- analyse: MAC, VLAN, errors
      |
      v
   Physical / NIC      <-- capture with tcpdump/Wireshark

Example analysis graph (congestion window from a real transfer):

Sequence number
    ^
    |   /‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
    |  /                                              .
    | /                                               .
    |/                               (packet loss)   .
    +------------------------------------------------------> Time
       <-- slow start --> <-- cong. avoid --> <-- recovery →

Protocol stack analysis helps diagnose performance issues, tuning TCP parameters, and understanding network behaviour.

---

Each of these mechanisms plays a vital role in modern TCP/IP stacks, balancing efficiency, fairness, and throughput.