Another important IPSec concept to consider is the mode in which you would have it participate on the network. The two IPSec modes are referred to as Transport mode and Tunnel mode. Transport mode allows for IPSec connections between 2 or more systems, used on demand, as the policy that you have created dictates. For example, you might specify that when any computer on the HR subnet attempts to communicate with the HR server, all traffic must be encrypted and signed. Or, you might specify that when any computer in the organization accesses the employee benefits database, all communication should be signed only. The level of granularity that you wish to use is up to you, just remember that the more complex your policies, the more difficult they may be to create and administer. Also remember that in order for a system to use IPSec in Transport mode, that system must use the IPSec protocol. Of the Microsoft operating systems, only Windows 2000 (and now XP) support these capabilities natively. Tunnel mode functions differently, and with a different purpose in mind. It creates a secured (and/or authenticated) tunnel between to endpoints. The idea is that traffic that passes through the tunnel in encrypted on one end and decrypted on the other, a scenario that would most commonly apply to moving between two private networks via an unsecured network such as the Internet. The benefits of using IPSec in Tunnel more are many. Firstly, it is relatively simple to set up (especially compared to Transport mode). Secondary, clients or servers on either side on the tunnel need not have any concept of IPSec or how it works. They simply send traffic destined for a system on the other side of the tunnel, and when it hits the first server the packets are encrypted, and subsequently decrypted on the other end. As you have probably considered, this does mean that the packets do travel over parts of both local networks in an unencrypted state. Whether that works for you will depend upon your specific security needs. Use a combination or tunnel mode and transport mode if it better meets your needs.
Many other documents also spend an incredible amount of time and energy explaining the internal processes that IPSec goes through in attempting to negotiate security. While this is certainly important, it is also easy to get bogged down in details and forget exactly what is going on. Essentially, if two computers want to use IPSec for the purpose of securing direct communication between them, they will each have to be configured with an IPSec policy. These policies are configured via local or group policy in Windows 2000, and will be discussed shortly. The policies need to work out the necessary details such that computers understand not only which protocols need to be secured, but also what key settings need to be used, authentication mechanisms, and so forth. The downside of IPSec is that there are many places where your configuration can be wrong. However, if you have everything mapped out properly in advance, the actual configuration should be fairly straightforward. Imagine if I have configured policies on two systems, and done it correctly. When the first system (say the client) attempts to communicate using IPSec with the second (say the server), they will first need to agree on a variety of factors including the key length to be used, security protocols, and something called the Security Parameter Index (SPI – this is what uniquely distinguishes one secured session from another). All of these things constitute something called a Security Association, the basis for the IPSec communication. You may have heard of something called the Internet Security Association and Key Management Protocol (ISAKMP) – this is the protocol used to actually establish security associations. There are actually two phases of security association negotiation, Phase I and Phase II. Phase I is responsible for the initial setup of a secure and authenticated channel between the two systems. Phase II is responsible for the actual security and negotiation of the data transfer, and the process of refreshing session keys used during the process. At the end of the Phase I SA lifetime (which is configurable), any Phase II SAs that exist may still exist, as they have their own lifetimes. However, if a Phase II data transfer were received without a Phase I SA in place, data transfer would be rejected.