Tuning the Linux Kernel TCP Stack for High-Throughput AI Workloads

Transferring multi-gigabyte or terabyte model checkpoints and training datasets across cluster nodes requires substantial network throughput. Default Linux kernel TCP settings are optimized for general web traffic, which severely caps performance below 100Gbps datacenter link speeds. Small default socket buffer sizes and loss-based congestion control algorithms throttle network throughput prematurely.

The Problem: Bandwidth-Delay Product and Buffer Starvation

The Bandwidth-Delay Product (BDP) determines the volume of unacknowledged data that must remain in transit to fully saturate a network link. On a 100Gbps network with 1ms round-trip latency, the calculated BDP requirement is approximately 12.5MB:

$$\text{BDP} = \frac{100 \text{ Gbps} \times 0.001 \text{ s}}{8} = 12.5 \text{ MB}$$

If sysctl receive and send socket buffers are restricted to default caps of 4MB or 6MB, TCP sender windows stall while waiting for acknowledgments (ACKs), leaving available network bandwidth underutilized.

Persistent Sysctl Configuration

The following script writes optimized TCP kernel parameters to /etc/sysctl.d/99-ai-network-tuning.conf and applies them across the host setup immediately.

#!/bin/bash
# Apply high-throughput TCP settings for AI/ML Nodes
set -euo pipefail
SYSCTL_CONF="/etc/sysctl.d/99-ai-network-tuning.conf"
cat << 'EOF' | sudo tee "$SYSCTL_CONF" > /dev/null
# 1. Socket buffer maximum caps (64MB)
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
# 2. TCP buffer auto-tuning ranges (min, default, max)
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
# 3. Fair Queueing + BBR congestion control
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
# 4. NIC backlog queue length
net.core.netdev_max_backlog = 30000
# 5. TCP Window Scaling (RFC 1323)
net.ipv4.tcp_window_scaling = 1
EOF
echo "Applying sysctl settings..."
sudo sysctl --system
echo "Current congestion control algorithm:"
sysctl net.ipv4.tcp_congestion_control

Kernel Parameter Breakdown

• net.core.rmem_max / net.core.wmem_max = 67108864: Increases the maximum socket read and write buffer cap to 64MB, providing high-BDP network connections with sufficient memory headroom to sustain in-flight data volumes.
• net.ipv4.tcp_rmem / net.ipv4.tcp_wmem: Configures kernel auto-tuning limits (minimum, default, and maximum buffer sizes). The Linux kernel dynamically adjusts buffer allocations per socket connection up to 64MB without consuming unnecessary memory on idle connections.
• net.ipv4.tcp_congestion_control = bbr: Replaces traditional loss-based CUBIC with BBR (Bottleneck Bandwidth and RTT). Unlike CUBIC, which reduces TCP congestion window sizes upon detecting packet loss, BBR models actual network capacity to maintain maximum throughput on high-speed links.
• net.core.default_qdisc = fq: Configures Fair Queueing packet pacing. FQ prevents BBR flows from transmitting micro-bursts of traffic that can overflow network switch buffers.
• net.core.netdev_max_backlog = 30000: Expands the network interface ingress queue depth at the driver layer, mitigating packet drops under heavy burst traffic conditions.
• net.ipv4.tcp_window_scaling = 1: Enables TCP window scaling extensions (RFC 1323), allowing window sizes to exceed the legacy 64KB boundary.

Persisting these parameter configurations within /etc/sysctl.d/ ensures settings remain active across platform reboots. On 100Gbps network links, proper TCP buffer scaling combined with BBR pacing delivers consistent line-rate data transfers between distributed training nodes.