Skip to content

VPN and tunneling protocols

The MITM article ended by pointing at encrypting the whole path to a trusted gateway as the direct defense against a hostile network in between — which is exactly what VPN basics already covered structurally: tunneling, split versus full tunnel, and what a VPN protects against. That article closed with a specific, named gap: it deliberately hadn't covered "how the encryption itself is negotiated, which protocol a given deployment should choose, or how WireGuard's approach differs from IPsec's." This article is that comparison.

Every VPN protocol solves the same two problems — negotiate a shared secret without ever transmitting it, and use that secret to authenticate and encrypt every packet afterward — but they differ enormously in how much machinery that takes, and that difference shows up directly in configuration complexity, performance, and how much attack surface the protocol exposes.

IPsec: a protocol suite built into the network layer itself

IPsec isn't one protocol but a suite, standardized across several RFCs (RFC 4301 defines its overall architecture, already cited in the VPN basics article), operating at the network layer rather than riding on top of it as an ordinary application would. It has two main pieces that solve different halves of the problem:

  • IKE (Internet Key Exchange) handles the negotiation phase: authenticating both sides to each other and agreeing on the cryptographic keys and algorithms everything afterward will use, itself a multi-message exchange comparable in purpose to the TLS handshake — different protocol, same underlying job of turning "we haven't talked before" into "we share a secret key."
  • ESP (Encapsulating Security Payload) is what actually wraps and encrypts each IP packet once IKE has finished. RFC 4301 describes ESP as providing confidentiality alongside "integrity and data origin authentication" — it's the piece doing the packet-by-packet work the tunneling diagram illustrated generically.

IPsec also formalizes the distinction between tunnel mode, where the entire original IP packet — header included — is encrypted and wrapped inside a new outer packet (the shape VPN basics already showed, and the mode almost every site-to-site and remote-access VPN actually uses), and transport mode, where only the payload is encrypted and the original IP header stays intact and visible. RFC 4301 requires every host-to-host implementation to support both, but reserves transport mode mainly for host-to-host protection on a network the endpoints already control directly — a security gateway providing remote access is expected to support tunnel mode specifically, since that's the mode that actually protects a client crossing an untrusted network in between.

The practical reputation IPsec has earned matches this architecture: it's mature, standardized long enough to be implemented in nearly every router, firewall, and operating system's kernel without third-party software, and correspondingly complex to configure correctly — IKE negotiation alone involves multiple phases and a long list of parameters (encryption algorithm, hashing algorithm, Diffie-Hellman group, and more) that have to match exactly on both ends, and a mismatch anywhere in that list fails the negotiation with an error that rarely names which specific parameter disagreed.

WireGuard: the same job, a much smaller codebase

WireGuard takes a deliberately different approach: instead of a suite of protocols each configurable with its own set of algorithm choices, it hardcodes one specific, modern set of cryptographic primitives and gives the administrator no algorithm choice to get wrong in the first place. Its own protocol documentation lists the fixed set directly: ChaCha20 for symmetric encryption authenticated with Poly1305, Curve25519 for the key exchange, and BLAKE2s for hashing.

That fixed cipher suite is also why WireGuard's handshake is so much shorter than IPsec's IKE negotiation: there's nothing to negotiate. The handshake is a two-message exchange built on the Noise protocol framework's Noise_IK pattern — the initiator sends one message containing an ephemeral public key and an encrypted static key, the responder replies with its own ephemeral key, and both sides derive the same session keys independently from that exchange. Contrast that directly with IKE's multi-phase negotiation above: WireGuard's entire key agreement is smaller than the parameter list IPsec administrators have to keep in sync by hand.

WireGuard runs entirely over UDP, and its interface model is exactly what VPN basics already showed with wg0 as one of the two common tunnel-interface names alongside tun0 — a WireGuard tunnel really is just a network interface from the operating system's point of view, with routing to and from it handled by the normal ip route table, the same table the VPN basics troubleshooting scenario inspected directly.

[Interface]
PrivateKey = <client-private-key>
Address = 10.8.0.5/24

[Peer]
PublicKey = <server-public-key>
Endpoint = 198.51.100.1:51820
AllowedIPs = 10.0.0.0/8

That configuration block is close to the entire client-side setup for a WireGuard tunnel — a private key, the peer's public key and address, and AllowedIPs, which does double duty as both a routing instruction and an access-control list: only traffic to 10.0.0.0/8 gets sent through this tunnel at all, which is the split-tunnel behavior VPN basics already described, expressed here as one line of configuration rather than a separate routing decision layered on afterward. Setting AllowedIPs = 0.0.0.0/0 instead produces full-tunnel behavior, routing everything through the peer.

TLS-based tunnels: reusing a handshake the reader already knows

A third family of VPN technology builds its tunnel directly on top of TLS instead of a purpose-built VPN handshake — the same TLS already covered in depth in the TLS handshake article and used for HTTPS. Once a TLS session is established between client and server, the tunnel simply carries arbitrary IP traffic through that already-encrypted, already-authenticated channel instead of an HTTP request and response.

The practical advantage this family of tunnels has over IPsec and WireGuard is almost entirely about network traversal rather than cryptography: TLS conventionally runs over TCP port 443, identical to ordinary HTTPS traffic, which means a TLS-based VPN tunnel often passes through restrictive firewalls and captive portals that block unrecognized UDP ports outright, because from the outside it's indistinguishable from a browser visiting an HTTPS site. IPsec and WireGuard, by contrast, use their own distinct ports and protocol behavior, both of which some restrictive networks specifically block.

Comparing the three side by side

IPsec WireGuard TLS-based tunnel
Layer / integration Network layer, often in-kernel Network layer, kernel module or userspace Application layer, on top of TLS
Cipher agility Configurable suite (IKE negotiates it) Fixed, hardcoded modern suite Whatever TLS cipher suites the endpoint supports
Handshake complexity Multi-phase IKE negotiation Two-message Noise_IK handshake Standard TLS handshake
Typical transport UDP (IKE and ESP) UDP TCP (usually port 443)
Firewall traversal Often blocked by strict egress filtering Often blocked by strict UDP filtering Passes almost anywhere HTTPS does
Configuration surface Large — many negotiable parameters Small — a handful of keys and addresses Depends on the surrounding application

None of these is universally "best" — the right choice depends on what the deployment actually needs. A network engineer standardizing site-to-site links between offices with full control over the firewalls on both ends has little reason to fight IPsec's configuration surface for functionality WireGuard delivers with far less of it. A team supporting remote employees who might be behind hotel or corporate firewalls that block anything unfamiliar has a genuine, concrete reason to prefer a TLS-based tunnel specifically for its ability to blend in as ordinary HTTPS traffic — not because the encryption is any stronger, but because a tunnel that can't establish a connection at all protects nothing.

Practical scenario: a WireGuard tunnel that won't establish from one hotel network

An employee's WireGuard client fails to connect only when tethered to a specific hotel's Wi-Fi, while the identical configuration on the identical laptop connects normally from home and from every other network tried so far.

sudo wg show

interface: wg0
  public key: (hidden)
  private key: (hidden)
  listening port: 51820

peer: (hidden)
  endpoint: 198.51.100.1:51820
  allowed ips: 10.0.0.0/8
  latest handshake: (none)
  transfer: 0 B received, 0 B sent

latest handshake: (none) confirms the tunnel interface exists and is configured, but no handshake has ever completed — the client's first message never got a reply. Since WireGuard runs over UDP on port 51820 by default, and the identical configuration works from other networks, the next reasonable check is whether this specific network's outbound filtering treats that UDP port differently than everywhere else:

nc -u -vz -w3 198.51.100.1 51820

nc: connect to 198.51.100.1 port 51820 (udp) failed: Connection timed out

The timeout, specific to this network and this UDP port, points at outbound UDP filtering on the hotel's guest network — a common restriction on networks that only expect to see standard web traffic leaving. This matches exactly the trade-off named in the comparison above: WireGuard's UDP transport is efficient and simple everywhere it's allowed through, and unreachable on precisely the kind of restrictive network that only permits traffic resembling ordinary HTTPS. The practical fix, without abandoning the tunnel entirely, is falling back to a TLS-based VPN option for this specific network, or configuring WireGuard's endpoint on a nonstandard port that isn't specifically filtered — the underlying protocol choice, not a misconfiguration, is what's actually being diagnosed here.

Practice exercises

  1. Using the comparison table above, explain which protocol a company should choose for a permanent site-to-site link between two offices where both firewalls are fully under the company's own control, and justify the choice against the alternatives.
  2. Explain what AllowedIPs = 0.0.0.0/0 in a WireGuard configuration does to a client's routing, in terms of the full-tunnel versus split-tunnel distinction from VPN basics.
  3. A remote employee reports their VPN works from home but not from their corporate office network, which has a strict egress firewall permitting only TCP port 443 outbound. Using this article's comparison, explain which of the three tunnel families is most likely to succeed on that network, and why.

Every protocol in this article assumes the tunnel's two endpoints — the client and the gateway — can be trusted once the tunnel is up, exactly the assumption Zero Trust opens by challenging directly: a device inside the tunnel is still just a device, and being inside it proves nothing about what that device should actually be allowed to reach.

Sources