cisspCommunication and Network Security· 11 min read· 17 May 2026
Secure Protocols: IPSec, TLS, SSH, and When to Use Each
Secure communication protocols are the technical mechanisms that protect data in transit. For the CISSP exam, Domain 4 tests whether candidates understand how the major secure protocols work, what security properties they provide, and when to use each. The exam also tests knowledge of insecure protocols and their secure replacements — a pattern that appears in multiple question types.
IPSec: Network-Layer Security Protocol Suite
IPSec (Internet Protocol Security) is a suite of protocols that provides security at the IP network layer (Layer 3). Because it operates below the application layer, IPSec is transparent to applications — any application that sends IP traffic can benefit from IPSec without modification.
IPSec uses two protocols to provide security services.
Authentication Header (AH) provides data origin authentication, data integrity, and anti-replay protection. It does not provide confidentiality (encryption). AH creates a cryptographic hash of the entire IP packet (including the IP header) and includes it in the AH header. This allows the recipient to verify that the packet has not been modified in transit and genuinely originated from the claimed source. However, because AH authenticates the IP header, it is incompatible with Network Address Translation (NAT) — NAT modifies IP headers, breaking the AH integrity check.
Encapsulating Security Payload (ESP) provides confidentiality (encryption), data origin authentication, data integrity, and anti-replay protection. ESP is the more commonly used IPSec protocol because it provides both confidentiality and integrity. ESP encrypts the payload (the data portion) of the IP packet using symmetric encryption (AES). Unlike AH, ESP can be used with NAT because it does not authenticate the outer IP header in tunnel mode.
IPSec operates in two modes:
Transport mode encrypts only the payload of the original IP packet, leaving the original IP header intact. The original source and destination IP addresses are visible. Transport mode is used for host-to-host communications where the two endpoints are the actual source and destination. It provides security for the data but not for routing information.
Tunnel mode encapsulates the entire original IP packet (header + payload) in a new IP packet with a new outer header. The original IP header is encrypted as part of the inner packet. The outer header contains the VPN gateway addresses, not the original source and destination. Tunnel mode is used for site-to-site VPNs (gateway-to-gateway) and remote access VPNs. It provides security for the complete original packet, including the original IP addressing information.
For the exam: IPSec AH provides integrity and authentication (no encryption). IPSec ESP provides encryption plus integrity and authentication. Transport mode for host-to-host; tunnel mode for VPN (entire packet protected). Tunnel mode is the correct choice for VPNs.
IKE: Internet Key Exchange
Before IPSec can protect traffic, the two endpoints must agree on cryptographic algorithms and exchange keys. This is accomplished through the Internet Key Exchange (IKE) protocol.
IKE Phase 1 establishes a secure, authenticated channel (the IKE Security Association) between the two endpoints. This can use either Main Mode (more secure, six message exchanges) or Aggressive Mode (three message exchanges, faster but reveals identity information).
IKE Phase 2 uses the Phase 1 channel to negotiate the IPSec Security Associations (one for each direction of traffic) that will protect the actual data traffic.
For the exam: IKE negotiates the cryptographic parameters for IPSec. Main Mode is more secure than Aggressive Mode.
TLS: Transport Layer Security
TLS (Transport Layer Security) is the successor to SSL (Secure Sockets Layer). SSL is deprecated and should not be used due to known vulnerabilities (POODLE, BEAST, DROWN). The exam expects you to know that when a question mentions SSL, it is describing a deprecated, vulnerable protocol.
TLS provides confidentiality (encryption), integrity (MAC), and authentication (certificate-based server verification, optional client verification). It is the security protocol used for HTTPS (HTTP over TLS), secure email (SMTPS, IMAPS), and many other application protocols.
TLS 1.3, released in 2018, made significant improvements over TLS 1.2. TLS 1.3 removes support for weak and obsolete cryptographic algorithms (RC4, DES, 3DES, MD5, SHA-1, RSA key exchange, DH with static keys). TLS 1.3 requires perfect forward secrecy (PFS) for all connections, using ephemeral Diffie-Hellman (ECDHE) for key exchange. This means that the session key is generated fresh for each session and is not stored — compromising the server's private key after the fact cannot decrypt previously recorded sessions. TLS 1.3 also improves handshake performance: the handshake requires only one round trip (1-RTT) instead of TLS 1.2's two round trips.
For the exam: SSL is deprecated and vulnerable. TLS 1.3 is the current standard. Key improvements in TLS 1.3 include mandatory perfect forward secrecy and removal of weak algorithms. If a question asks about encrypting HTTPS traffic, the answer is TLS (not SSL).
SSH: Secure Shell
SSH (Secure Shell) is a cryptographic network protocol for secure remote login, command execution, file transfer (SFTP), and tunnelling. It is the secure replacement for Telnet (unencrypted remote login) and FTP (unencrypted file transfer).
SSH provides: confidentiality (encrypted session), integrity (MAC on all data), and authentication (password or public key-based).
SSH authentication methods include:
Password authentication: the user provides a password that is transmitted over the encrypted SSH connection. Weaker than key-based authentication because passwords can be brute-forced.
Public key authentication: the user's SSH client proves possession of the private key corresponding to a public key registered on the server. This is significantly more secure than password authentication and is preferred for administrative access. Key pairs should be generated with strong algorithms (Ed25519 or RSA-4096).
SSH security hardening best practices: disable password authentication (require key-based authentication), disable root login (require users to log in as non-root and escalate privileges), change the default port from 22 (reduces automated scanning noise, though this is security through obscurity and not a real security control), implement fail2ban or similar tools to block IP addresses after repeated authentication failures, and use SSH certificates (rather than individual key files) in large deployments.
SSH tunnelling (also called SSH port forwarding) allows TCP connections to be tunnelled through an SSH connection. Local port forwarding redirects a local port to a remote destination through the SSH server. Remote port forwarding exposes a local service to the remote SSH server. Dynamic port forwarding creates a SOCKS proxy. SSH tunnelling can be used for legitimate purposes (accessing internal services through a bastion host) but can also be used to bypass security controls (tunnelling prohibited protocols through the permitted SSH connection).
For the exam: SSH replaces Telnet (unencrypted). SFTP replaces FTP (unencrypted). Key-based SSH authentication is preferred over password authentication. SSH tunnelling can bypass security controls.
Secure vs Insecure Protocol Pairs
A common exam question type asks which secure protocol should replace a specified insecure protocol. The standard replacements are:
Telnet (unencrypted remote login) → SSH (encrypted remote login). Telnet transmits all data including credentials in plaintext and should never be used for administrative access.
FTP (unencrypted file transfer) → SFTP (SSH File Transfer Protocol) or FTPS (FTP over TLS). FTP transmits credentials and data in plaintext. SFTP uses SSH for transport. FTPS uses TLS.
HTTP (unencrypted web) → HTTPS (HTTP over TLS). HTTP exposes all data including form submissions and session cookies to interception.
SNMP v1/v2 (unencrypted network management) → SNMPv3. Early SNMP versions use plaintext community strings for authentication. SNMPv3 adds authentication and encryption.
RDPwithout TLS (unencrypted or weakly encrypted remote desktop) → RDP with Network Level Authentication (NLA) and TLS. Unencrypted RDP exposes the entire remote desktop session.
Exam Tip
IPSec tunnel mode protects the entire packet including original IP header — use for VPNs. Transport mode protects only the payload — use for host-to-host. TLS replaces SSL (which is deprecated and vulnerable). SSH replaces Telnet and FTP. The exam frequently presents a scenario using an insecure protocol and asks which secure protocol should replace it — know all the secure/insecure pairs.
// PRACTICE_THIS_DOMAIN
Test your knowledge on Communication and Network Security
AI-generated practice questions mapped to this domain. Get instant explanations and track your progress.