Security and the TCP/IP stack

Each layer of TCP/IP has security mechanisms, protocols, and applications. This tip describes the more popular ones that are associated with each layer of TCP/IP.

This tip is in response to a reader question submitted to our network administration and security expert Michael...


Dear Michael,
What type of security is associated with each level of the OSI model?

The following is Michael's reply:

That's a very good question. I am going to use the TCP/IP model as a reference as it is the protocol of the Internet and networks around the world. TCP/IP is a four layer model. If you want to learn more about TCP/IP check out this Cisco page as it has lots of good information.

The four layers of the TCP/IP model are, from bottom to top, 1) physical, 2) network, 3) transport, and 4) application as shown below.

Each layer has security mechanisms, protocols, and applications. I will list some of the more popular ones that are associated with each layer of the TCP/IP stack.

  1. Physical – The physical layer comprises layer one and two of the OSI model
    1. Packet Filters – A packet filter is designed to set between the internal and external network. As packets enter or leave the network, they are compared to a set of rules. This determines if they are passed, rejected, or dropped. A router ACL is an example of a packet filter.
    2. NAT – NAT (Network Address Translation) is a means of translating addresses. Most residential high speed Internet users use NAT. It provides security as it hides the internal address from external networks.
    3. CHAP - CHAP (Challenge Handshake Authentication Protocol) is an authentication protocol that is used as an alternative to passing clear text usernames and passwords. CHAP uses the MD5 hashing algorithm to encrypt passwords.
    4. PAP – While PAP (Password Authentication Protocol) may not be the best security mechanism at the physical layer, it does provide some protection as it requires a user to present a username and password. Its Achilles heel is that it transmits this information in clear text.

Learn about protocols related to TCP/IP

Protocols related to TCP/IP include the User Datagram Protocol (UDP), which is used instead of TCP for special purposes. Other protocols are used by network host computers for exchanging router information. These include the Internet Control Message Protocol (ICMP), the Interior Gateway Protocol (IGP), the Exterior Gateway Protocol (EGP), and the Border Gateway Protocol (BGP).

  1. Network – The network layer matches up to layer three of the OSI model.
    1. PPTP – PPTP (Point to Point Tunneling Protocol) was developed by a consortium of vendors including Microsoft and 3Com. Its purpose is to provide data encapsulation. Security for PPTP is provided by Microsoft Point-to-Point Encryption.
    2. L2TP – This VPN protocol is used for security and was based on PPTP and L2F.
    3. IPsec – IPsec is used to protect IP packets and defend against network attacks. It uses cryptographic-based protection services, security protocols, and dynamic key management. IPsec has two basic configurations AH (Authenticated Header) and ESP (Encapsulated Secure Payload).


  2. Transport – The transport layer relates to layer four and five of the OSI model.
    1. SSL – SSL (Secure Sockets Layer) is a protocol independent technology that enables users to ensure security for data that is exchanged over the Internet. Read more about it here.
    2. TLS – This protocol is similar to SSL. The TLS (Transport Layer Security) protocol is a layered approach to data security that consists of several sub-protocols.
    3. More IPsec - ESP is found at the transport layer, as it encapsulates the data for security and privacy.


  3. Application – The application layer includes some of layer five and all of layer six, and layer seven of the OSI model.
    1. RADIUS – RADIUS (Remote Authentication Dial-In User Service) is the most widely used dialup authentication protocol in the world. It offers authentication and authorization to dial-up network users.
    2. TACACS – Whatis.com defines TACACS (Terminal Access Controller Access Control System) as "an older authentication protocol common to UNIX networks that allows a remote access server to forward a user's logon password to an authentication server to determine whether access can be allowed to a given system." Read more about it here.
    3. Kerberos – Kerberos was invented by MIT to be a strong authentication protocol. It uses tickets to validate user's rights to objects. It provides encryption, privacy and data integrity.
    4. S-MIME - S/MIME (Secure / Multipurpose Internet Mail Extensions) is a protocol designed to secure e-mail. It secures clear-text email by adding digital signatures and encryption.
    5. Virus scanners – While this may be the final item on my list it is by no means any less important than any of the other security mechanism discussed. Virus scanners play an important part of security!

Actually, there is a name for what I have been describing here. It is called layered security. Layered security is an approach that builds defense in depth. This principle can significantly reduce the risk of attack or loss of CIA (Confidentiality, Integrity, and Availability), as it increasing the costs and resources required by an attacker to break into a network or host. While no network or host can ever be 100% secure, defense in depth can greatly reduce the risk of successful attack.

Dig Deeper on Network Infrastructure

Unified Communications
Mobile Computing
Data Center