Chapter 3

Guide to Operating System Security

Chapter 3

Security Through Authentication and

Encryption

2 Guide to Operating System Security

Objectives

Explain encryption methods and how they are used

Describe authentication methods and how they are used

Explain and configure IP Security Discuss attacks on encryption and

authentication methods


Encryption

Uses a secret code to disguise data Makes data unintelligible to everyone except

intended recipients Protects data from attackers using a sniffer Uses cryptography Typically involves a key and an algorithm


Encryption Methods (Continued)

Stream cipher and block cipher Secret key Public key Hashing Data encryption standard (DES) RSA encryption


Encryption Methods (Continued)

Pluggable authentication modules (PAMs) Microsoft Point-to-Point Encryption (MPPE) Encrypting File System (EFS) Cryptographic File System (CFS)


Stream Cipher and Block Cipher

Stream cipher Every bit in a stream of data is encrypted

Block cipher Encrypts groupings of data in blocks Typically has specific block and key sizes


Secret Key

Keeps encryption key secret from public access, particularly over a network connection

Uses symmetrical encryption (same key to encrypt and decrypt)


Public Key

Uses public key and private key combination (asymmetric encryption)

Public key can be communicated over an unsecured connection


Hashing

Uses one-way function to mix up message contents Scrambles message Associates it with a unique digital signature Enables it to be picked out of a table

Often used to create a digital signature Hashing algorithms work on only one side of a

two-way communication


Typically Used Hashing Algorithms

Message Digest 2 (MD2) Message Digest 4 (MD4)

MS-CHAP v1 MS-CHAP in Windows Server 2003

Message Digest 5 (MD5) Secure Hash Algorithm 1 (SHA-1)


MS-CHAP v1 or MS-CHAP Encryption


Data Encryption Standard

Developed by IBM; refined by the National Bureau of Standards

Originally developed to use a 56-bit encryption key

New version: 3DES (Triple DES) Hashes data three times Uses a key of up to 168 bits in length


Using DES with IPSec in Windows Server 2003


Advanced Encryption Standard

Adopted by U.S. government to replace DES and 3DES

Employs private-key block-cipher form of encryption

Employs an algorithm called Rijndael


RSA Encryption

Uses asymmetrical public and private keys along with an algorithm that relies on factoring large prime numbers

The algorithm uses a trapdoor function to manipulate prime numbers

More secure than DES and 3DES Used in Internet Explorer and Netscape

Navigator


Pluggable Authentication Modules

Can be installed in UNIX or Linux OS without rewriting and recompiling existing code

Enable use of encryption techniques other than DES for passwords and communications on a network


Microsoft Point-to-Point Encryption

Used by Microsoft operating systems for remote communications over PPP or PPTP

Uses RSA encryption Basic encryption (40-bit key) Strong encryption (56-bit key) Strongest encryption (128-bit key)


Encrypting File System

Set by an attribute of Windows OSs that use NTFS

Protects folder/file contents on hard disk Enables user to encrypt contents of folder/file so it

can only be accessed via private key code by user who encrypted it

Employs DES for encryption Uses a registered recovery agent


How to Configure EFS

As an advanced folder attribute By using cipher command in Command

Prompt window


Configuring EFS as an Advanced Folder Attribute


Cipher Command-Line Parameters


Cryptographic File System

File system add-on available as open source software for UNIX and Linux systems

Enables encryption of disk file systems and NFS files


Summary of Encryption Techniques (Continued)

continued…


Summary of Encryption Techniques (Continued)


Authentication

Process of verifying that a user is authorized to access particular resources

Typically associated with logon process Validates both user account name and

password before giving access to resources Often uses encryption techniques to protect

user names and passwords


Authentication Methods (Continued)

Session authentication Digital certificates NT LAN Manager Kerberos Extensible Authentication Protocol (EAP) Secure Sockets Layer (SSL)


Authentication Methods (Continued)

Transport Layer Security (TLS) Secure Shell (SSH) Security token


Session Authentication

Ensures packets can be read in correct order Provides a way to encrypt the sequence order

to discourage attackers


Digital Certificate

Set of unique identification information typically put at the end of the file or associated with a computer communication

Shows that the source of the file or communication is legitimate

Typically encrypted by a private key and decrypted by a public key

Issued by a certificate authority


Digital Certificate Contents

Version Certificate serial number Signature algorithm identifier Name of issuer Validity period Subject name Subject public key information


NT LAN Manager

Form of session authentication and challenge/response authentication compatible with all Microsoft Windows operating systems

Challenge/response authentication Hashes an account’s password Uses a secret key


Kerberos

Employs private-key security and use of tickets that are exchanged between the client who requests logon and network services access and the server, application, or directory service that grants access


Kerberos Configuration Options


Extensible Authentication Protocol

Multipurpose authentication method used on networks and in remote communications

Can employ many encryption methods (DES, 3DES, public key encryption, smart cards, and certificates)

Typically provides an authentication communication between a computer and a server used to authenticate computer’s access


Secure Sockets Layer

Service-independent; broad uses fore-commerce, HTTP, HTTPS, FTP, SMTP, and NNTP

Developed by Netscape Uses RSA public-key encryption Most commonly used form of security for

communications and transactions over the Web


Transport Layer Security

Modeled after SSL Uses private-key symmetric data encryption

and TLS Handshake Protocol


Secure Shell

Developed for UNIX/Linux systems to provide authentication security for TCP/IP applications, including FTP and Telnet


Using Secure Shell


Security Token

Physical device, often resembling a credit card or keyfob, used for authentication

Communicates with an authentication server to generate the password, using encryption for exchange of password-generating information


Advantages of Security Token

User does not have to memorize password Value of password only lasts as long as the

communications session; new password is created next time the security token is used



IP Security (IPSec)

Set of IP-based secure communications and encryption standards developed by the IETF

Protect network communications through IP Elements that enable security measures

Authentication header Encapsulating Security Payload (ESP)


IPSec Security Roles


Authentication Header (AH)

Ensures integrity of a data transmission Ensures authentication of a packet by enabling

verification of its source


Specific Fields in AH

Next header Payload length Reserved Security Parameter Index (SPI) Sequence number Authentication Data


Encapsulating Security Payload (ESP)

Encrypts packet-based data Authenticates data Generally ensures security and confidentiality

of network layer information and data within packet


Specific Fields in ESP

Security Parameter Index (SPI) Sequence number Payload data Padding Pad length Next header Authentication date


Attacks on Encryption and Authentication


Guidelines for Resisting Attacks

Use strong passwords Use strongest forms of authentication and

encryption permitted by OS Use longest encryption keys possible Inventory encryption and authentication

methods used by OS; close any holes Have administrators use personal accounts

with administrative privileges (rather than use administrative account directly)


Summary

Encryption methods and how operating systems use them

How systems authenticate one another How to configure Kerberos authentication

logon security How to use IP security to keep your TCP/IP

network secure Typical methods attackers use to defeat

encryption and authentication

Documents

Chapter 3