What Is VPN Tunnel Encryption? (IPsec Tunnel)
An IPsec tunnel protects traffic between two gateways by wrapping each complete inner IP packet in an ESP packet and adding a new, visible outer IP header for routing. With IKEv2 and a suitable authenticated-encryption method, the gateways can establish keys and protect data. The tunnel does not hide its endpoints or encrypt the outer header.
Think of it like putting a sealed parcel inside a new shipping box. The contents are protected between the two gateways, while the new box still needs an address so the network can deliver it. That is the core idea behind IPsec tunnel encryption. I focus first on what must match at both ends, then on how to check that traffic really uses the tunnel.
Plan the IPsec Tunnel and Select Compatible Proposals
An IPsec tunnel links two security gateways and protects traffic selected by their policies. RFC 4301 defines the IPsec architecture; RFC 4303 defines ESP; RFC 7296 defines IKEv2; and RFC 4106 specifies AES-GCM for ESP. These standards describe different parts of the system, so a sound setup must align them.
In tunnel mode, ESP protects the entire original IP packet, including its inner header. The gateways add a new outer IP header so routers can forward the packet to the remote gateway. That outer header is not encrypted. The gateway addresses, and some routing information, remain observable along the path.
Before configuring anything, write down the design at both sites:
- Public or otherwise reachable address of each gateway.
- Local and remote protected subnets.
- Routes and any NAT between peers.
- Expected traffic, including large transfers.
- IKE and ESP algorithms supported by both endpoints.
- The path MTU, meaning the largest packet the route can carry without fragmentation.
A security association (SA) is the set of negotiated parameters and keys used to protect traffic in one direction. IKEv2 creates an IKE SA for management and Child SAs for protected data. A traffic selector describes which local and remote addresses or subnets a Child SA covers.
Prefer authenticated encryption such as AES-GCM when both peers support a compatible configuration. It provides confidentiality and integrity in one algorithm. The peers must agree on proposals, authentication, selectors, and lifetimes. Use strong peer authentication and unique credentials. Do not rely on a weak or deprecated algorithm simply because it makes negotiation succeed.
AES-GCM has an important key-use rule: never reuse a nonce with the same key. RFC 4106 defines a four-octet salt and an eight-octet explicit IV for its ESP nonce construction. A 16-octet authentication tag is common, but tag length is configured or negotiated. Confirm the actual settings on both devices rather than assuming defaults match.
| Design item | What to compare at both peers | Why it matters |
|---|---|---|
| IKE version and proposal | IKEv2, encryption, integrity where applicable, and key exchange settings | A mismatch can prevent IKE negotiation |
| ESP proposal | AES-GCM support, tag settings, and lifetimes | These define how data packets are protected |
| Traffic selectors | Local and remote subnets in opposite order | Mismatches can leave intended traffic outside the SA |
| Authentication | Certificates or a strong, unique shared credential | Confirms the peer identity |
| Routing | Routes and policies to protected networks | Negotiation alone does not direct traffic into the tunnel |
The first practical takeaway is simple: document the two sides as a pair. A tidy proposal and selector table can reveal errors before they become confusing traffic failures.
Check Endpoints, Firewall Traversal, and Path MTU
The gateways need a path to exchange both IKE control messages and protected ESP traffic. Firewalls and NAT devices can affect that path. At the same time, the tunnel adds packet overhead, so the usable inner packet size is smaller than the underlying path can carry.
IKEv2 uses UDP destination port 500 for initial exchanges. When NAT traversal is used, IPsec traffic is encapsulated in UDP, typically on port 4500. Native ESP uses IP protocol number 50. It is not TCP or UDP port 50. Permit the traffic required by the chosen design through intervening firewalls, and check both directions where policy requires it.
If NAT is present, confirm that both gateways support the needed NAT traversal behavior and that the firewall permits the resulting UDP traffic. Do not assume that opening UDP 500 alone allows data traffic to pass. Native ESP may be blocked by a network policy even when IKE succeeds.
Calculate the tunnel’s packet-size limits
The tunnel adds overhead for ESP, the outer IP header, and possibly UDP encapsulation. ESP overhead also depends on the selected algorithm, tag, and packet padding. For that reason, there is no single safe MTU or MSS value for every IPsec tunnel.
Path MTU is the largest packet size that can travel across a route without fragmentation. Check the path between the gateways and account for the full encapsulated packet, not just the inner network’s MTU. Where NAT-T is used, include its UDP header as well. Avoid setting a value by habit without testing the actual path.
A common trap is that small pings work while large transfers stall. The tunnel may establish and pass short packets, yet larger packets may exceed the tunnel path MTU. Check whether Path MTU Discovery (PMTUD) works end to end. If needed, apply a suitable TCP MSS clamp on the relevant gateway, based on measurements and device guidance. A clamp helps TCP; it does not fix every kind of traffic.
Configure IKEv2, ESP Tunnel Mode, and Routing
Configuration has four linked parts: define the peers, negotiate matching protection, permit the required packets, and direct selected traffic into the tunnel. IKEv2 manages key exchange and Child SAs; ESP tunnel mode carries protected data. Correct routing and policy are just as important as successful negotiation.
Use this sequence:
- Define the peers. Enter each gateway’s tunnel endpoint and set strong, matching authentication. For a shared secret, use a unique, high-entropy value and configure it securely on both devices. Do not send it in plaintext.
- Set IKE and Child SA proposals. Match supported IKEv2 settings and ESP settings at both ends. Prefer authenticated encryption such as AES-GCM where supported. Align lifetimes and any required key exchange settings.
- Set traffic selectors. Enter the protected local and remote subnets in opposite order at the two peers. Check for overlapping selectors or networks that already use the same address range.
- Permit and route traffic. Allow UDP 500 and, when used, UDP 4500. Allow native ESP, protocol 50, if the design uses it. Install the required routes and policies so intended traffic enters the tunnel.
- Prevent accidental fallback. Confirm that protected traffic does not silently leave through an ordinary route if the tunnel fails. The exact control depends on the gateway platform.
A successful IKE exchange proves that the peers can negotiate. It does not prove that the intended application traffic is selected, routed, encrypted, or able to cross the tunnel at its required packet size.
Two practical deployment scenarios
These are illustrative configurations, not measured case reports. They show how the same checks apply to common site-to-site designs.
Two small offices: Each gateway protects one local subnet to the other office’s subnet. I would first verify that the subnets do not overlap, then set matching selectors and routes. If either office is behind NAT, I would confirm UDP 4500 traversal and test traffic in both directions.
A remote site with large file transfers: The peers may establish SAs and pass short tests while large transfers fail. I would compare the path MTU with the encapsulated packet size, inspect PMTUD behavior, and test a suitable MSS clamp for TCP if required. I would not treat an established tunnel as proof that the file-transfer path is sound.
Verify Security Associations, Traffic, and Rekeying
Verification means checking both control-plane status and real protected traffic. An IKE SA shows that the gateways negotiated management keys; a Child SA shows that they negotiated a data-protection relationship. Neither alone proves that the right packets use it or that large packets pass.
Use a staged test:
- Confirm the IKE SA is established and the expected Child SA is active.
- Check that the local and remote traffic selectors match the intended subnets.
- Generate traffic between hosts in those subnets, in both directions.
- Check ESP packet and byte counters at both gateways. Counters should rise when test traffic crosses.
- Test small packets, then larger packets and the applications that matter.
- Review logs for proposal mismatches, authentication errors, selector rejection, replay issues, or MTU-related failures.
- Observe a rekey and, if the design requires it, test failover.
If a ping succeeds but an application stalls, investigate packet size, routing, host firewalls, and selector coverage before changing encryption settings. If counters rise on one gateway but not the other, check the return route, peer policy, and firewall path. Keep a record of the negotiated algorithms and the observed test results.
| Observation | Likely area to inspect | Next check |
|---|---|---|
| No IKE SA | Reachability, UDP 500/4500, authentication, or IKE proposal | Review peer logs and firewall rules |
| IKE SA but no Child SA | ESP proposal, selectors, or Child SA settings | Compare both peers’ data proposals and subnets |
| SA active, counters unchanged | Routing, policy, or traffic selectors | Send traffic between protected hosts and inspect routes |
| Small traffic works, large traffic stalls | Path MTU, PMTUD, or fragmentation handling | Test larger packets and review MSS settings |
| One direction works | Return route, stateful firewall, or peer policy | Check counters and routing at both ends |
| Failure during rekey | Lifetimes or proposal mismatch | Compare negotiated settings and rekey logs |
The key test is not simply “tunnel up.” It is whether the intended traffic passes in both directions, with the expected protection, through normal packet sizes and rekey events.
Conclusion and FAQ
IPsec tunnel encryption uses ESP to protect complete inner IP packets between gateways, while a new outer IP header remains visible for routing. Reliable results depend on matching IKEv2 and ESP settings, correct selectors and routes, firewall traversal, and measured MTU behavior. Verify real traffic, not only SA status.
What is an IPsec tunnel?
It is a connection between security gateways that uses IPsec to protect selected IP traffic as it travels between them.
What does ESP tunnel mode encrypt?
It protects the entire original IP packet as the ESP payload. The new outer IP header remains visible so routers can deliver the packet.
Does an IPsec tunnel hide the gateway addresses?
No. The outer header contains addresses used to route traffic between tunnel endpoints. IPsec tunnel mode does not make those endpoints invisible.
What is the difference between IKEv2 and ESP?
IKEv2 negotiates keys and security associations. ESP carries and protects the selected data traffic.
Which ports and protocols must a firewall allow?
IKEv2 uses UDP 500 for initial exchanges. NAT traversal uses UDP 4500. Native ESP is IP protocol 50, not a TCP or UDP port.
Why can a tunnel be up while applications fail?
IKE and Child SAs can be active even when routes, selectors, or firewall rules are wrong. Added packet overhead can also cause large packets to stall.
What are traffic selectors?
They define which local and remote addresses or subnets a Child SA protects. Both peers must configure compatible selectors.
Is AES-GCM suitable for ESP?
AES-GCM is an authenticated-encryption option defined for ESP by RFC 4106. Confirm both peers support matching settings, and never reuse a nonce with the same key.
How do I check that traffic uses the tunnel?
Send traffic between protected hosts, then check SA status, selectors, routes, and ESP byte and packet counters at both gateways. Test both directions and larger packets.
Should I set one fixed MTU or MSS value for every tunnel?
No. Overhead and path limits vary. Measure the path, account for ESP and any UDP encapsulation, test PMTUD, and use an appropriate MSS clamp when needed.
(This article was written by one of our staff writers, Garrett Vance. Visit our Meet the Team page.)
