IPSEC VPN Breakdown

Phase I | Main mode | Securing the Channel

At this point the peers authenticate and the SA is negotiated between gateway peers. This will set the secured channel for our traffic. Preshared key is matched between peers here within the three exchanges of algorithm, dh group and peer identity. At this point, messages are sent out to the receiving end that you can capture on the device. These messages are sending out signs of contact and information to be handled and matched by the peer. Whether they are answered or not hints towards specific issues with establishing the secure tunnel. By the end of message 6, phase 2 should also be completed showing the tunnel as up and able to pass traffic between local and remote subnets.

ISAKMP Negotiations

MM_WAIT_MSG2 Initiator Initial SA details sent to remote peer. Encryption, hash, dh group sent to create initial contact. Awaits response from peer.

·       Other end is not responding.

·       Possible remote end does not have ISAKMP enabled

·       Upd 500/upd 4500 not enabled

MM_WAIT_MSG3 Receiver Receiver is sending back its SA details to initiator.

·       Hang ups here may also be due to mismatched device vendors,

·       A router with a firewall in the way, or even ASA version mismatches

MM_WAIT_MSG4 Initiator Initiator is sending the Pre-Shared-Key hash to its peer. Initiator will stay at MSG4 until it gets a PSK back from its peer.

·       If the receiver is missing a tunnel group 

·        PSK missing

·       MM_WAIT_MSG5 Receiver Receiver is sending its PSK hash to its peer. Receiver does not yet check if PSK hashes match. If receiver has a tunnel-group and PSK configured for this peer it will send the PSK hash to the peer.

·       PSK unavailable

·       NAT-T on when should be off

MM_WAIT_MSG6 Initiator Initiator checks if PSK hashes match. If PSK keys match, Initiator becomes MM_ACTIVE and lets receiver know of match.

·       PSK mismatch If PSK doesn’t match

·       NAT-T on when should be off

·       If the state goes to MSG6 then the ISAKMP gets reset

o     Phase 1 finished but phase 2 failed. Check that IPSEC settings match in phase 2 to get the tunnel to stay at MM_ACTIVE.

Phase I | Aggressive mode | Building the Channel

It should be mentioned that main mode is chosen superior to aggressive, since main mode sends the peer SA over an encrypted channel, while aggressive mode does not, unless used with digital certificates or a public key. Fortigates have the option to accept Phase I negotiation in aggressive mode. Within aggressive mode, 3 messages are sent as opposed to the 6 that you saw above. Therefore, meaning less packets are sent resulting in a quicker connection. Customers may decide to use this because both peers have dynamic external IP addresses. This however results in some limitations.

Phase II | IKE Phase II or Quick mode | Phase II Negotiations

At this point the encrypted channel has been established. Yet the negotiation of settings is not complete. At this point, the internal network’s SA will be negotiated and established, then renegotiated periodically for extra security. At the time of the SA expiration, quick mode will renegotiate it using the DH group from phase I. If PFS (perfect forward secrecy) is used, a new DH group is exchanged. The more DH group exchanges, the more bloated the values produced become as they increase exponentially. This exhausts the device further, resulting in higher CPU use.

At this point, internal traffic between local and remote subnets can pass traffic. Packets will now be encrypted and decrypted.

Fortigate Specific Logs

ike 0:isa_server:11: responder: main mode get 1st message...

ike 0:isa_server:11: responder:main mode get 2nd message...

ike 0:isa_server:11: responder: main mode get 3rd message...

ike 0:isa_server:11:134: responder received first quick-mode message

ike 0:isa_server:11:ph2_isa_server:134: matched phase2