Virtual Private Network (VPN) Protocols

Several Software Development Companies started developing VPN technologies in the mid-1990s, and their protocols were vendor-specific. Toward the late 1990s, VPNs converged toward the IPSec standard. Today, most VPN vendors are using IPSec as the basic protocol in their products.

RFC 6040 is an Internet Engineering Task Force (IETF) standardization of the Security Architecture for the Internet Protocol (aka IPSec). It specifies how VPNs should work across platforms, thereby achieving vendor interoperability. The standard does not prohibit more advanced functionality from being added by any particular vendor, but it does set forth minimum standards by which compliant devices will be able to communicate to form VPNs. It is necessary for the settings on both sides of a VPN to match exactly.

The most common VPN protocols in use today are IPSec, PPTP, and L2TP over IPSec, and SSL VPNs. Let’s take a look at each of these.

1. IPSec

IPSec was released in 1998, after years of design and debate among security specialists and product manufacturers. It represents a sort of compromise among various different interests. IPSec was designed to provide confidentiality through encryption, authentication of endpoints, and secure key management. It provides different ways of doing these things, largely due to the design debates that preceded its release. The IPSec parameters that are used by the endpoints are negotiated in the Security Association (SA). With two built-in security protocols and two “modes” of operation, four different combinations of protocols and modes are possible—and you’d better make sure they match on both endpoints, or your VPN won’t work.

There are two types of configurations when using IPSec VPNs: transport mode and tunnel mode. Transport mode only encrypts the payload data. Although transport mode could be considered “faster” depending on what crypto protocols you are using (because there’s less decryption required with transport mode), most crypto hardware today processes in silicon and is fast enough that you cannot tell the difference between tunnel mode and transport mode except in very demanding/high-load environments or extremely latency sensitive environments. For that reason, transport mode is not very common any more with today’s high-speed equipment.

Tunnel mode is primarily what you’ll find in use in today’s networks, so it’s what we’ll focus on in this section. Tunnel mode encrypts both the IP header and the payload data, and in turn has two separate ways it can do this—Authentication Header (AH) and Encapsulating Security Payload (ESP), both of which add a new IP header that contains the tunnel endpoints while encrypting the IP header containing the real source and destination of the packet.

Nearly all operating systems or VPN-related products that have IPSec support have the ability to create IPSec tunnel-mode tunnels. Because most products can create this type of tunnel, its most significant advantage is interoperability. Due to different interpretations of authentication and crypto processing, creating the IPSec tunnel relationship can be a time-consuming, manual process. All settings on both sides of a tunnel must match exactly in order for the tunnel to form. You will need to ensure that the products you select to do a job are capable of satisfying all necessary objectives.

IPSec tunnel mode is often used for L2L gateway-to-gateway connections such as business partner links and branch office connections in Software Development Companies. This is because of its flexibility, optimization, and vendor compatibility, and also because the parameters of the connection do not often change, unlike other sorts of client connections. Often the connection parameters for building the tunnel between the devices must be manually created, and although this works well when linking sites together, it would be impractical to support this for clients whose connection details (such as endpoint IP address) change often. Vendors that have chosen this tunnelling protocol solution have worked around the need for manual configuration in a variety of ways, but for any type of non-static tunnel, implementations are specific to the technology platform and are not cross-vendor interoperable.

Remote access VPN clients also use IPSec tunnel mode, but they typically require the client to load connection software, such as a VPN client or local installation package that creates a virtual network adapter, which can be preloaded with the client-side configuration requirements such as authentication options and group membership.

Authentication Header (AH)

As shown in Figure, AH tunnel mode encapsulates an IP packet with an AH and an IP header and signs the entire packet for integrity and authentication.

Encapsulating Security Payload (ESP)

As shown in Figure, ESP tunnel mode encapsulates an IP packet with both an ESP header and an IP header and an ESP authentication trailer. Comparison of AH and ESP The main difference between AH and ESP is that ESP provides encryption (that’s the “E” in the name) and AH does not. As with transport mode, as mentioned above, the lack of encryption was designed to speed up traffic flow on the comparatively slow network devices of the late 1990s when the IPSec protocol was first defined. Tunnel mode and ESP, which provide encryption, were meant for faster devices. Of course, today’s devices are so fast that the encryption rarely makes a difference any more.

Because of export restrictions, it may sometimes be necessary to use AH if you are deploying a VPN in a country where you are not allowed to use encryption. Otherwise, you’ll probably opt to use ESP in tunnel mode in most situations, unless you are configuring a VPN with a third party whose device you don’t control, and they have chosen a different protocol and mode combination.

2. PPTP

There are a number of other protocols that have been developed over the years and are still part of many products today. The most common of these protocols is Microsoft’s Point to-Point Tunnelling Protocol (PPTP). PPTP is still used quite a bit in the industry because it is easy to deploy, flexible, and supported by most operating systems today. PPTP was initially deployed in 1998 as part of Windows NT 4.0 and was immediately pounced on by the press because of its horrible initial security model. This has largely been corrected in Windows 2000 and 2003, but PPTP’s reputation will likely be forever marred by the initial mistakes. At least through Windows 2008 R2, Microsoft still has PPTP capabilities, and some people recommend using it as a “quick and dirty” way to get remote access to a network, but in general it is less secure than other methods and should only be used, with caution, to allow access to networks that are not carrying mission-critical or confidential data.

3. L2TP over IPSec

L2TP over IPSec is the result of combining the best parts of PPTP from Microsoft and Layer Two Forwarding Protocol (L2F) from Cisco, dropping both of those protocols’ encryption protocols and using IPSec instead as the encryption solution in Software Development Companies. L2TP over IPSec uses IPSec transport mode and has the advantage of being a PPP-based tunnel, which allows two things: protocols other than TCP/IP can easily be supported in the tunnel, and the operating systems can create a known connection object that can be used to address the tunnel (this is particularly important in Microsoft operating systems). These options are significant if the operating system design allows the use of multiple protocols.

Although L2TP over IPSec gives both the client and the server more flexibility, it creates overhead in the tunnel environment that could be argued is unnecessary, particularly if the environment only uses TCP/IP. However, many organizations that deploy VPNs consider the native Windows support of L2TP over IPSec to be a significant advantage. Typically, L2TP over IPSec is used for client-to-server connectivity because the connection parameters can handle dynamically changing clients. If your network environment requires the use of protocols other than TCP/IP, L2TP can also be used for gateway-to-gateway connections. Since both IPSec and L2TP are defined within the IETF standard, there are more vendors that support this solution, although there are still many more supporting IPSec tunnel mode.

4. SSL VPNs

Modern SSL VPN implementations can exactly match (or exceed) the functionality of a software client-based VPN for remote access. When it comes to user remote access, many products leverage SSL-secured links to corporate applications via some type of authenticated-access portals, which are commonly called “published links.” This approach has three major advantages:

• Nearly every client has the needed software loaded by default—the Internet browser. No additional software is needed.
• Most firewalls support SSL, and no additional protocols or ports need be opened to support this type of connection.
• In many cases, remote users simply need to perform predictable tasks, such as checking their e-mail or running a specific application, and many of those are already web-based.

Many organizations have turned to SSL VPNs for remote access because IPSec client based tunnels have extensive computing overhead (they require the administrative burden of creating, distributing, and maintaining a connection file local to the PC), they can be expensive, and many vendors have problems with their proprietary extensions.

Traditionally, a Client-Based VPN supplemented the client’s full network stack via a software-based virtual network adapter, which supported different network protocols and ports. Many SSL VPN platforms can now support this type of functionality via a “virtual client” (often a Java software package) that is loaded on the fly when a client session is authenticated. The virtual adapter works in almost exactly the same way as the VPN client software that would be running on the endpoint in a client-based IPSec remote access VPN scenario, but the virtual adapter does not require an administrator to preload anything.

It is likely that SSL will become the de facto standard for encrypting most externally accessed corporate communications in Software Development Companies. Generally, it is the only method used for accessing cloud-based applications; it is easier to deploy and maintain than client-based VPNs, and is completely flexible since it is a basic component of both web browsers and web servers alike.