Helpful context:

You click a link. Fifty milliseconds later, a webpage appears. That gap - negligible enough that you barely notice it - is one of the most elaborate sequences of coordinated events in modern engineering. A packet left your machine, was inspected and forwarded by a dozen routers across continents, triggered a TLS cryptographic negotiation, and returned with a response, all before your coffee cooled by a single degree. This post is about what happened in those 50ms.

The Problem That Made the Stack Necessary

In the 1970s, ARPANET worked, but it was fragile. A nuclear strike on a single relay station could sever communication. The U.S. Department of Defense funded research into a network architecture where there was no single point of failure - where packets could route around damage autonomously. The result was the IP/TCP stack, finalized around 1974 by Vint Cerf and Bob Kahn.

The insight was layering. Each layer does one job and hands off to the next. The physical layer moves bits. The link layer moves frames between directly connected devices. The network layer (IP) routes packets across the planet without caring what’s underneath. The transport layer (TCP/UDP) adds reliability or speed characteristics. The application layer (HTTP, DNS, TLS) defines what the data means.

This separation of concerns is why a video call over Wi-Fi uses the same TCP/IP stack as a data center fiber link. The layers are contractually independent.

IP: The Envelope That Crosses the Planet

Every packet that travels the internet carries an IP header containing a source address and a destination address. IPv4 addresses are 32-bit integers written in dotted decimal - 93.184.216.34. CIDR notation like 192.168.1.0/24 means the first 24 bits identify the network; the remaining 8 bits are available for 256 host addresses on that subnet.

Routers make forwarding decisions one hop at a time. A router looks at a packet’s destination IP, consults its routing table, and forwards the packet toward the best next hop. No router knows the full path end-to-end - just the next step. This is why the internet can route around failures: routing protocols (BGP between autonomous systems, OSPF within them) continuously exchange topology information and recompute paths.

The TTL (Time To Live) field in the IP header starts at some value (often 64 or 128) and is decremented by each router. If it reaches zero, the packet is dropped and an ICMP “time exceeded” message is sent back to the sender. This prevents routing loops from trapping packets in infinite cycles. traceroute exploits this: it sends packets with TTL=1, gets ICMP replies from the first hop, then TTL=2, and so on, mapping the entire path.

TCP: Reliability on an Unreliable Foundation

IP delivers packets on a best-effort basis. Packets can be dropped, reordered, or duplicated. If you’re downloading a file and every packet arrived scrambled, that would be unusable. TCP’s job is to build a reliable, ordered byte stream on top of IP’s chaos.

Before a single byte of application data flows, TCP performs a three-way handshake:

Client                Server
  |--- SYN ---------->|   seq=1000
  |<-- SYN-ACK -------|   seq=5000, ack=1001
  |--- ACK ---------->|   ack=5001

This establishes initial sequence numbers on both sides and confirms that both can send and receive. Only after the handshake does the HTTP request go out - which means every new TCP connection costs at least one round-trip time (RTT) before a byte of useful data is exchanged. On a 100ms RTT connection (think US to Europe), that’s 100ms of pure overhead before your request even starts.

Sequence numbers give every byte a unique ID. The receiver can detect gaps, reorder out-of-order segments, and discard duplicates. Acknowledgements tell the sender which bytes have been received. If an ACK isn’t received within a timeout, the sender retransmits.

Flow control prevents a fast sender from drowning a slow receiver. The receiver advertises a window size in each ACK - the number of bytes it’s prepared to buffer. The sender cannot have more than that many unacknowledged bytes in flight at once.

Congestion control is more subtle. The receiver’s buffer is one bottleneck; the network itself is another. TCP slow start begins every new connection with a tiny congestion window (initially one segment, roughly 14KB today) and doubles it each RTT. This probing continues until a packet is dropped - the signal that the network is saturated. Then TCP backs off. The algorithm is called AIMD: additive increase, multiplicative decrease.

The Slow Start Problem (and Why It Killed HTTP/1.1)

Slow start was designed for long-lived connections - bulk data transfers where the ramp-up period is a small fraction of the total. But web pages are different. Each HTTP/1.1 request fetches a small resource (a script, an image, a CSS file) over a new connection, or at best a keep-alive connection that may have gone idle. A connection that goes idle resets its congestion window. A new request must ramp up from scratch.

For short connections - 20 packets or fewer - TCP often never leaves slow start. You’re paying the ramp-up cost for every single resource. This is the fundamental inefficiency HTTP/2 was designed to fix.

A Discomfort Check: TCP Is Reliable, But Not What You Think

“TCP guarantees ordered delivery” is technically true and practically confusing. If TCP receives packet 3 before packet 2, it buffers packet 3 and waits for packet 2 to be retransmitted. From the application’s perspective, packets always arrive in order. But this creates head-of-line blocking: if one packet is lost in a multiplexed HTTP/2 stream, all other streams stall waiting for that one packet to be recovered. HTTP/2 multiplexes multiple requests over one TCP connection - great in theory, but a single dropped packet blocks every stream because TCP’s ordering guarantee applies at the connection level, not the stream level.

This is not a bug in TCP. It’s doing exactly what it promised. The problem is that multiplexing wants per-stream ordering, but TCP only knows about connection-level ordering.

UDP: When Speed Beats Reliability

UDP is TCP without the guarantees. No handshake, no acknowledgements, no flow control, no ordering. A UDP datagram either arrives or it doesn’t. The application must decide what to do.

This is the right tradeoff in several scenarios. DNS queries are tiny and idempotent - if the reply is lost, the client retries immediately. The overhead of a TCP handshake would dwarf the query itself. Live video calling cannot wait for retransmitted packets; a delayed frame is more disruptive than a corrupted one, so WebRTC uses UDP with its own lightweight reliability layer. Online games need sub-50ms updates; they’d rather skip a position update than wait for recovery.

The unspoken truth: most transport innovation of the last decade is UDP-based protocols implementing their own TCP-like features, because they can do it with more control and less overhead than the kernel’s TCP stack.

HTTP: The Language of the Web

HTTP defines what clients and servers say to each other after TCP is set up. It’s request-response: the client sends a request with a method, URL, headers, and optional body; the server responds with a status code, headers, and body.

HTTP/1.1 (1997): text-based protocol, one outstanding request per TCP connection at a time. You can send multiple requests over one connection (keep-alive), but they’re serialized - you must wait for the response before sending the next. Browsers worked around this by opening 6 parallel TCP connections per origin, meaning 6 concurrent slow starts. Hacky, but effective enough for a decade.

HTTP/2 (2015): binary framing, single TCP connection, full multiplexing. Multiple requests and responses interleave over one connection in numbered streams. Header compression (HPACK) cuts overhead. Server push lets the server send resources before the client asks. The single connection means one slow start instead of six, and HPACK means headers don’t repeat on every request. This alone delivered 20 - 50% latency improvements for many workloads.

The catch: that one TCP connection’s head-of-line blocking problem. HTTP/2 solved application-layer HoL blocking but exposed transport-layer HoL blocking. On lossy mobile networks, a single dropped packet stalls the entire connection.

HTTP/3 (2022): runs over QUIC instead of TCP. QUIC is a UDP-based protocol that reimplements TCP’s reliability features per-stream rather than per-connection. A dropped packet for stream 3 doesn’t block stream 7. QUIC also integrates TLS 1.3, collapsing the handshake from 2 RTTs to 1 (or even 0 for resumed connections). Google has run QUIC in production since 2013; it now carries roughly 25% of all internet traffic.

TLS: The Padlock Explained

When you see HTTPS, a TLS handshake has happened before any HTTP request. In TLS 1.3 (the current standard):

  1. Client sends ClientHello with supported cipher suites and a key share (Diffie-Hellman public value).
  2. Server responds with its certificate (identity, signed by a CA) and its own key share.
  3. Both sides independently compute the same shared secret from the key exchange. All subsequent data is encrypted with symmetric keys derived from this secret.

The certificate chain establishes trust: your server’s certificate is signed by an intermediate CA, which is signed by a root CA embedded in your operating system or browser. If the chain validates and the hostname matches, the padlock appears.

TLS 1.3 reduced the handshake to 1 RTT (down from 2 in TLS 1.2). Combined with QUIC, HTTP/3 connections can complete the TLS handshake and send the first HTTP request in a single round trip.

Cloud Connections: L4 vs L7 Load Balancers

Load balancers operate at different layers of the stack, and the distinction matters enormously.

An L4 load balancer (AWS Network Load Balancer) sees TCP segments. It can distribute connections across backend servers but cannot inspect HTTP headers. It’s extremely fast - it doesn’t parse application data - but it can’t make routing decisions based on URLs, cookies, or request content. It’s appropriate when you need maximum throughput with simple routing.

An L7 load balancer (AWS Application Load Balancer, Nginx, Envoy) sees HTTP requests. It can route /api/* to one backend cluster and /static/* to another. It can terminate TLS, inspect headers, perform authentication, rate-limit, and rewrite URLs. It’s far more powerful but adds latency and processing cost.

CDNs (Cloudflare, Akamai, AWS CloudFront) terminate TCP connections at edge servers close to the user. This is not just for caching - it drastically reduces latency. A TCP connection from New York to a London origin has 70ms RTT. If Cloudflare terminates that connection at a New York PoP (point of presence), the user’s TCP handshake is 5ms. The CDN then has a persistent, pre-warmed TCP connection to the London origin server. The user experience improves even for cache misses.

gRPC, widely used for microservice-to-microservice communication, runs over HTTP/2. This means it needs an L7 load balancer that understands HTTP/2 framing. An L4 load balancer will route the TCP connection once and all streams go to the same backend - no per-request load balancing.

The Critique: TCP Is 1974 Technology Running 2024 Networks

TCP was designed for wired networks with stable, low-loss characteristics. It interprets packet loss as congestion and backs off. Modern mobile networks drop packets due to signal interference, not congestion - but TCP reacts the same way, unnecessarily throttling throughput.

The congestion control algorithm is conservative by design. In a data center where RTT is 0.1ms and the link is lossless, TCP’s slow start and additive increase are absurdly cautious. Protocols like RDMA (Remote Direct Memory Access) used in high-performance computing skip TCP entirely and go directly to the network card.

QUIC is the most significant effort to move beyond TCP’s limitations without abandoning the internet’s installed base. Because it runs over UDP and lives in user space rather than the kernel, it can be updated without waiting for OS vendors - Google ships QUIC updates in Chrome. The fundamental question for the next decade is whether QUIC becomes the default internet transport the way TCP replaced everything that came before it.

eBPF is a related revolution. It lets you run sandboxed programs inside the Linux kernel, including in the network stack, without writing kernel modules. Tools like Cilium use eBPF to implement network policies, load balancing, and observability at wire speed. The XDP (eXpress Data Path) hook can process packets before they even hit the kernel’s normal network stack - enabling software-defined networking performance that approaches hardware.

Future Outlook

HTTP/3 adoption continues to grow. As of 2024, roughly 30% of the top million websites support it. The remaining friction is middlebox compatibility - enterprise firewalls, legacy proxies, and some ISPs block or interfere with UDP on non-DNS ports, which QUIC needs.

The combination of QUIC and eBPF may eventually make today’s L4/L7 distinction irrelevant. eBPF programs can implement load balancing, DDoS mitigation, and routing at the network card level, without the overhead of proxying through a userspace process. Cloudflare and Facebook already use this in production.

Concept Key Point
IP Connectionless, best-effort packet delivery; each router makes hop-by-hop decisions
TTL Prevents routing loops; traceroute exploits it to map paths
TCP 3-way handshake Costs 1 RTT before data flows; slow start adds more
TCP slow start Starts small, doubles congestion window; hurts short connections disproportionately
Head-of-line blocking Lost TCP packet stalls all HTTP/2 streams on that connection
UDP No reliability; right for DNS, gaming, live video, QUIC
HTTP/1.1 Text, serialized requests, 6 parallel connections as workaround
HTTP/2 Binary, multiplexed streams, single connection, solves app-layer HoL
HTTP/3 / QUIC UDP-based, per-stream reliability, eliminates transport-layer HoL
TLS 1.3 1-RTT handshake; 0-RTT for resumed connections
L4 vs L7 LB L4 sees TCP connections; L7 sees HTTP requests and can route by content
CDN TCP termination Edge PoP absorbs user’s high-RTT connection; fast pre-warmed link to origin

Read Next: