Motivation for Separation Kernels

Inzemamul HaqueDepartment of Computer Science and Automation

Indian Institute of ScienceBangalore-560012

Email: inzemamul.haque@csa.iisc.ernet.in

AbstractThis article tries to figure out the motivation behindthe use of separation kernels as secure systems. In this article theauthor has tried to draw a complete picture in the mind of thereader that how the security issues in different types of systemsled to the development of separation kernels.

Keywordskernel, Multics, operating systems, security, securitykernel, separation kernel, UNIX.

I. INTRODUCTION

In this article we try to find out the loopholes in thesecurity of various operating systems which led to differenttypes of secure kernels viz. security kernels, separation kernels.In section II, we shall try to define a secure operating system.And then in later sections we shall go through differentoperating systems in the course of the history of developmentof operating systems and analyze their security. In section III,IV, V, and VI, we shall discuss Multics, Unix, security kernelsand separation kernels respectively.

II. SECURE OPERATING SYSTEM

A secure operating system is an operating system whichimplements the reference monitor concept.

A reference monitor concept enforces a mandatory protec-tion system and giving the following guarantees:

Complete mediation: The system should ensure thatall security sensitive operations are mediated throughthe access control enforcement mechanism.

Tamperproof: The access control enforcement mecha-nism cannot be modified by untrusted processes.

Verifiable: The access control enforcement mechanismand the protection system should be small enough tobe subject to test and analysis.

III. HISTORY

MIT, General Electric and Bell Labs started the Multicsproject[3] in 1963. This project was not successful becauseof high ambitions of the project. Bell laboratories pulled outin 1969. Most of the fundamental operating system conceptsincluding security were developed in the course of this project.Finally Multics was bigger, slower and less reliable thanexpected. Multics development was terminated in 1985, butit was in use until 2000.

In mid-1970s, most of the researchers figured out that hav-ing two many features (like generality, performance, efficiencyand security) on the same system was difficult because of the

size of the system becoming too large. Hence the vendors andthe researchers started to choose one of the two directions -(i) generality and performance oriented systems with limitedsecurity and (ii) security oriented systems with reasonableperformance. The first one resulted in the development ofUNIX like systems and second one resulted in secure kernelslike security kernels and separation kernels.

As stated earlier Bell Labs left the Multics project in 1969.Ken Thompson and Dennis Ritchie(researchers from Bell Labswho also worked in Multics project) used their experiencefrom Multics project and developed the new operating systemUNIX, which was going to rule the world for almost nextfour decades. UNIX was developed to play a game on aPDP-7 computer. Hence it was very small in the beginning,however it grew in size in later decades because of its usein academics and industries. UNIX was simpler and smallerthan Multics. Hence it was easier to program and was betterin performance as compared to Multics. UNIX adopted manyof security features of Multics, but the security goals of UNIXwere different from Multics as it was a personal project. Weshall discuss the security problems and a few vulnerabilitiesof UNIX in next section.

In 1970s research was going on to build a secure operatingsystem. And then the Anderson report [1] came up with thereference monitor concept. Implementation of reference mon-itor concept came to be known as Security Kernels. The firstsecurity kernel was developed by MITRE in 1974 which hadless than 20 subroutines in less than 1000 source lines of code.Other systems which came out as a result of addressing thesecurity limitations of Multics were Secure CommunicationsProcessor (Scomp) [4] from Honeywell, the Gemini SecureOperating System (GSOS or GEMSOS) [8] from Gemini, theSecure Ada Target (SAT) etc. We shall discuss the problemswith security kernels in section V.

Then in 1981 John Rushby came up with the idea ofa separation kernel [7] which tried to address some of theproblems in security kernels. Separation kernels are discussedin section VI.

IV. UNIX

Earlier we talked about history of UNIX and found out thatsecurity was not one of the goals of UNIX. Hence there willbe some loopholes.

First of all UNIX is a discretionary access control system.A system in which the process which has the ownership ofthe object can change the permissions for other processes iscalled a discretionary access control system. Alternatively it

can be said that the protection state is at the discretion of theusers and any untrusted processes that they may execute. Hereif all users are non-malicious then we can ensure security butnowadays users run a variety of processes some of which maybe supplied through the attackers (may be some links in thee-mail).

UNIX authorizes operations on files by mediating file openfor read write and execute permissions. But this is not alwayspossible because some objects are not represented as files.Another problem is that by even having the read access for afile, a user process can perform a variety of operations on thefile besides reading by using fcntl or ioctl system call. UNIXdoes not mediate security sensitive objects such as networkcommunications.

UNIX has a command named chroot which is used to limita process to a subtree of a file system. An attacker can create/etc/passwd and /etc/shadow files in the subtree and can add anentry for root. The attacker can login as this root and escape thechroot environment by using the root access to kernel memory.

A. UNIX Security Analysis

As described in section II, a secure operating system shouldmeet the requirements of complete mediation, tamperproofand verifiable. Now we shall analyse UNIX under theserequirements.

Complete mediation - UNIX fails to control theaccess to information. As discussed above, UNIXpermits modification to file without having a writepermission for the file by using fcntl or ioctl. Wealso saw that UNIX does not mediate access to someof the system resources, e.g. in case of objects suchas network communications, UNIX does not provideauthorization at all.

Tamperproof - User-level processes can access andmodify kernel beyond system calls. It can installkernel modules to special file systems (e.g., /proc orsysfs). User process can have direct access to kernelmemory via the device file /dev/kmem.

Verifiable - The large size of UNIX makes the formalverification very difficult. The extensible nature ofthe UNIX system (by having new device drivers andkernel modules) makes it more difficult.

So finally we can say that UNIX fails to meet the require-ments of complete mediation, tamperproof and verifiability.Hence UNIX is not a secure system.

B. UNIX vulnerabilities

In this part we try to show the problems which can befaced if the system design does not focus on the security asone of the main goals.

Rootkits - Rootkits are the softwares which hide theexistence of malicious programs in the system. It isvery difficult to find the rootkit in the system.

Time-of-Check-to-Time-of-Use (TOCTTOU) - Thisis a big vulnerability in UNIX. Let us consider thefollowing example to understand this attack. A user

process opens a file /tmp/X for which it has the accesspermission. The operating system will first check theaccess permission for the process to open the file byusing access system call. And then execute the opensystem call but between the authorization and openingof the file, the user process changes the file X to asymbolic link for another file for which the access isdenied. In this way the attacker can access a file for thepermission is denied. These kind of attacks are calledtime-of-check-to-time-of-use (TOCCTOU) attacks.

V. SECURITY KERNELS

Security kernels are the systems which implement thereference monitor concept. These systems do not provide asmany features as provided by ordinary operating systems likeUNIX, windows etc. These systems are very simple in termsof functionality. By definition security kernels are secure asthe secure operating system is the one which implements thereference monitor concept. While mediation and tamperproofwere fundamental to the design of the security kernels butverifiability became the focus in building the security kernels.

The problem with the security kernels was the use oftrusted processes. The concept of trusted processes can beexplained by the example presented below.

Suppose there is a printer which has to print the jobs ofdifferent class of users. Assuming that we are using the Bell-LaPadula policy model [2], the printer should have the highestclassification because it has to print the jobs of all class ofusers. But there are two problems -

users cannot check the performance of their job asaccording to Bell-LaPadula model, lower class of usercannot read above its class.

when the printer finishes some job, it has to delete thefile from the spooler, but according to Bell-LaPadulamodel a user with higher classification cannot writeto the objects of a user who is below him.

To solve these problems, the concept of a trusted processwas introduced. A trusted process is the one which can violatethe policy and trusted by the system.

After the inclusion of the trusted processes in the systemthe trusted computing base becomes kernel with trusted pro-cesses. So it should be ensured that privileges granted to trustedprocesses should not be abused by the processes. Thereforenow both the security kernel and trusted processes need to beverified. But the problem is that there does not exist a formalmodel for such a combination of kernel and trusted processes.And also there are a lot of trusted processes which makesit more difficult to come up with such model. As Landwehrmentions

. . . in the final version of their model, Bell andLaPadula did include trusted processes. What isnot included in their exposition is a technique forestablishing when a process may be trusted.[6]

In a single word the main problem was verificationof system in presence of trusted processes. To solve theproblem of trusted processes, Rushby came up with the idea

of separation kernel [7]. The next section describes the detailsof a separation kernel.

VI. SEPARATION KERNELS

Separation kernel is a type of security kernel which par-titions the system into different partitions which are isolatedwith each other. There can be some explicit communicationchannels as given by the policy but no implicit communicationchannels. Different processes can run in different partitions.

Rushby solved the problem of printer spooler by taking theprinter spooler and the file system in different partitions withsome explicit communication channels. Here the security ismaintained by the isolation provided by the separation kernel.

ACKNOWLEDGMENT

Most of the material covered in this article, especially thesection III is taken from the book Operating System Security[5] by Trent Jaeger. The issues in security kernel because oftrusted processes described in section V are mostly from thepaper by Rushby [7].

The author would like to thank Prof. Deepak DSouza,Indian Institute of Science, Bangalore and Arnab Kundu,CAIR, DRDO for their valuable discussions on this topic.

REFERENCES[1] J. P. Anderson, Computer Security Technology Planning Study, Tech-

nical Report ESD-TR-73-51, The MITRE Corporation, Air Force Elec-tronics Systems Division, Hanscom AFB, Badford, MA, 1972.

[2] D. E. Bell and L. J. LaPadula, Secure COmputer Systems: UnifiedExposition and Mul- tics Interpretation, Technical Report ESD-TR-75-306, MITRE corporation, Bedford, MA, March 1976.

[3] F. J. Corbato and V. A. Vyssotsky, Introduction and Overview of MulticsSystem, In Proceedings of the 1965 AFIPS Fall Joint ComputerConference, 1965.

[4] L. J. Fraim, SCOMP: A Solution to the Multilevel Security Problem,IEEE Computer, 16(7):26-34, 1983.

[5] T. Jaeger, Operating System Security, Morgan & Claypool Publishers,2008.

[6] C. E. Landwehr, Assertions for Verification of Multilevel Secure MilitaryMessage Systems, ACM Software Engineering Notes. 5(3):46-47, July1980.

[7] J. Rushby Design and Verification of Secure Systems, In Proceedings ofthe Eighth ACM Symposium on Operating System Principles, pp. 12-21,Dec 1981.

[8] W. R. Shockley, T. F. Tao, and M. F. Thompson, An Overview ofthe GEMSOS class A1 Technology and Application Experience, InProceedings of 11th National Computer Security Conference, pp.238-245, October 1988.