IPSec VPN: What It Is and How It Works

What Is IPSec?

IPSec suite of VPN protocols includes Authentication Header (AH), Encapsulated Security Payload (ESP), Internet Security Association and Key Management Protocol (ISAKMP), and IP Payload Compression (IPComp).

• Authentication Header (AH): AH offers data origin authentication of IP packets (datagrams), guarantees connectionless integrity, and gives protection against replay attacks (thanks to the sliding window technique). The AH also offers significant authentications for both IP headers and upper-layer protocols.
• Encapsulating Security Payload (ESP): ESP is responsible for offering authentication, integrity, and confidentiality of data. ESP also provides payload confidentiality and message authentication within the IPSec protocol suite.
  In tunnel mode, it encapsulates the entire IP packet, while only the payload is protected in transport mode.
• Internet Security Association and Key Management Protocol (ISAKMP): ISAKMP is tasked with Security Associations (SAs) – a set of pre-agreed keys and algorithms used by parties when establishing a VPN tunnel. These include Kerberized Internet Negotiation of Keys (KINK) and Internet Key Exchange (IKE and IKEv2).
• IP Payload Compression (IPComp): IPComp is a low-level compression protocol that reduces the size of IP packets, thereby improving the communication levels between two parties. This is useful when communication is overly slow, for instance, congested links.
  IPComp doesn’t offer security and must be used with AH or ESP over VPN tunnels.

IPSec VPN Operation Modes

Let’s look at how the two IPSec VPN modes compare:

IPSec Tunnel Mode

VPN encryption in tunnel mode encapsulates every outgoing packet with new IPSec packets using ESP. Tunnel mode also uses AH to authenticate the server-side.

Hence, IPSec uses tunnel mode on secure gateways like a firewall that links the two communicating parties.

Transport Mode

Transport mode encrypts and authenticates the IP packets sent between two communicating parties.

As such, transport mode is often reserved for end-to-end communications between parties, considering it doesn’t change the IP header of the outgoing packets.

Cryptographic Algorithms for IPsec

IPSec relies on secure algorithms that conform to confidentiality, integrity, and authenticity.

They include:

• Authentication algorithms such as RSA, PSK, and Elliptic Curve cryptography.
• Symmetric encryption algorithms such as AES-CBC and GCM, HMAC-SHA, TripleDES, and ChaCha20-Poly1305.
• Key exchange algorithms such as Elliptic Curve Diffie-Hellman and the Diffie-Hellman key exchange.

How Does IPSec Work?

Below is a general step-by-step outline of how IPSec works.

Usually, the process starts with hosts (communicating parties) establishing that incoming or outgoing packets need to use IPSec.

If the packets trigger IPSec policies, then the process continues as follows:

• Negotiation and Key exchange: This step includes host authentication and policies to be used. In the first phase, the hosts create a secure channel. Negotiations are done using either the main mode (for greater security) or the aggressive mode (for faster IP circuit establishment).
  All hosts agree on an IKE for setting up the IP circuit in the main mode. In aggressive mode, the initiating host presents the IKE for setting up the IP circuit, and the other host agrees.
  In the second phase, the hosts negotiate and agree on the type of cryptographic algorithms to be used during the session.
• Transmission: This involves the exchange of data between the hosts. Usually, IPSec breaks data into packets before it’s sent over the network. The packets include several segments like the payload and headers. IPSec also adds trailers and other segments that contain authentication and encryption details.
• End of transmission: This is the last step, and it involves the termination of the IPSec secure channel. Termination occurs when data exchange is complete or the session has timed out. Cryptographic keys are also discarded.

IPsec VPN vs. SSL VPN

Besides IPSec VPN, many of the best VPN providers can also use SSL VPN to secure your connection over the internet. Depending on the level of security needed, VPN providers can implement both or choose one over the other.

SSL VPNs rely on the Transport Layer Security (TLS) protocol. Unlike IPSec, which works on the IP layer, TLS works on the transport layer. Thus, the security and applications of IPSec VPN and SSL VPN vary.

With IPSec VPN, your traffic is secure as it moves to and from private networks and hosts; in a nutshell, you can protect your entire network. Thus, IPSec VPN is reliable for IP-based uses and applications.

SSL VPN protects the traffic between remote users. In most cases, SSL VPNs work with hosts that support browser-based applications.