Cs33 Os Basics

7/27/2019 Cs33 Os Basics

1/61

SomenotesadoptedfromBryantandOHallaron


2/61

Role of The Operating System?Role of The Operating System?


3/61

ProcessesProcesses Def:Aprocessisaninstanceofarunningprogram.

Oneofthemostprofoundideasincomputerscience.

Notthesameasprogramorprocessor

Processprovideseachprogramwithtwokeyabstractions:

Logica contro ow

EachprogramseemstohaveexclusiveuseoftheCPU.

Eachprogramseemstohaveexclusiveuseofmainmemory.

HowaretheseIllusionsmaintained?

Processexecutions

interleaved

(multitasking)

Addressspacesmanagedbyvirtualmemorysystem


4/61

Logical Control FlowsLogical Control Flows

Each process has its own logical control f low

Process A Process B Process C

Time


5/61

Concurrent ProcessesConcurrent Processes Twoprocessesrunconcurrently(areconcurrent) if

theirflowsoverlapintime.

Otherwise,they

are

sequential.

Concurrent:A&B,A&C


Time


6/61

User View of Concurrent ProcessesUser View of Concurrent Processes Controlflowsforconcurrentprocessesare

physicallydisjointintime.

However,we

can

think

of

concurrent

processes

are

runnin in arallelwitheachother.



7/61

Context SwitchingContext Switching ProcessesaremanagedbyasharedchunkofOScode

calledthekernel

Control

flow

passes

from

one

process

to

another

via

a

contextswitch.

code

code

user code

erne co e

user code

Time

context switch

user code


8/61

Process: Traditional ViewProcess: Traditional View Process=processcontext+code,data,andstack

Programcontext:

Code,data,andstackstack

Processcontext

sharedlibraries

DataregistersConditioncodes

Stackpointer(SP)Program

counter

(PC) brk

read/writedata

readonlycode/dataPCKernelcontext:

VMstructuresDescriptortable

0brk pointer


9/61

Process: Alternative ViewProcess: Alternative View Process=thread+code,data,andkernelcontext

sharedlibrariesProgramcontext:Datare isters

Code,data,andkernelcontextThreadruntimeheap

read/writedata

ConditioncodesStackpointer(SP)

Programcounter

(PC) read

only

code/dataPC

r

0stack

SP Kernelcontext:

VMstructures

Descriptortablebrk pointer


10/61

Process with Two ThreadsProcess with Two Threads

Programcontext:

Thread1

sharedlibraries

atareg sters

Conditioncodes

Stackpointer(SP)Programcounter(PC)

o e, a a,an erne con exbrk

read/writedata

readonlycode/datastack

SPPC

0

Kernelcontext:VMstructures

Programcontext:Data re isters

Thread2

Descriptortable

brk pointer

Conditioncodes

Stackpointer(SP)Programcounter(PC)

stackSP


11/61

Threads vs. ProcessesThreads vs. Processes Threadsandprocesses:similarities

Eachhasitsownlogicalcontrolflow

Eachcan

run

concurrently

with

others

Eachiscontextswitched scheduled b thekernel

Threadsandprocesses:differences

,not

Processcontrol(creatingandreaping)ismoreexpensiveas

threadcontrol

Contextswitchesforprocessesmuchmoreexpensivethanfor

threads


12/61

Threads vs. Processes (contd.)Threads vs. Processes (contd.) Processesformatreehierarchy

Threadsforma oolof eers

Eachthread

can

kill

any

other

Mainthread:firstthreadtoruninaprocessProcesshierarchy Threadpool

P0

T1

T2T4

sh sh sh

sharedcode,dataandkernelcontext

fooT5 T3


13/61

Virtual Memory (Previous Lectures)Virtual Memory (Previous Lectures) Programsrefertovirtualmemoryaddresses

movl (%ecx),%eax 000

Conceptuallyverylargearrayofbytes

Eachbytehasitsownaddress

Systemprovidesaddressspaceprivatetoparticularprocess

Wheredifferentprogramobjectsshouldbestored

Allallocationwithinsin levirtualaddresss ace

Butwhyvirtualmemory?

FFF


14/61

Problem 1: How Does Everything Fit?Problem 1: How Does Everything Fit?

64bitaddresses: Physicalmainmemory: xa y e ew ga y es

Andtherearemanyprocesses.


15/61

Problem 2: Memory ManagementProblem 2: Memory Management

Physicalmainmemory

Whatgoesstackheap

.text

rocessProcess2Process3

x . rocessn


16/61

ProblemProblem 33: How To Protect: How To Protect

Physicalmainmemory

Processi

Processj


17/61

Problem 4: How To Share?Problem 4: How To Share?

Physicalmainmemory

Processj


18/61

Solution: Level Of IndirectionSolution: Level Of Indirection Eachprocessgetsitsownprivatememoryspace

Solvesthe revious roblems

Virtualmemory

Physical

memory

Process1

mappingV rtua memory

Processn


19/61

Address SpacesAddress Spaces Virtualaddressspace

SetofN=2nvirtualaddresses:{0,1,2,3,,N1}

Physicaladdress

space

=

Cleandistinctionbetweendata(bytes)andtheir

Eachobjectcannowhavemultipleaddresses

Everybyte

in

main

memory:

onephysicaladdress,one(ormore)virtual

addresses


20/61

A System Using Physical AddressingA System Using Physical Addressing

0:

Mainmemory

1:

CPU

2:3:4:

Physicaladdress(PA)

6:7:

8: ...

M1:

Usedinsimplesystemslikeembedded

Data

word

microcontrollersindeviceslikecars,elevators,and

digitalpicture

frames


21/61

A System Using Virtual AddressingA System Using Virtual Addressing

0:1:

Mainmemory

CPUChipMMU

2:

3:4:5:

Physicaladdress

(PA)CPU

Virtualaddress

(VA)

7:

8:..

.

M1:

Dataword

Usedin

all

modern

desktops,

laptops,

workstations

MMUchecks

the

cache


22/61

Why Virtual Memory (VM)?Why Virtual Memory (VM)? Efficientuseoflimitedmainmemory(RAM)

UseRAMasacacheforthepartsofavirtualaddressspace

somenoncac e partsstore on is

some(unallocated)

non

cached

parts

stored

nowhere

Kee onl activeareasofvirtualaddresss aceinmemor

transferdatabackandforthasneeded

Simplifiesmemory

management

for

programmers

Eachprocessgetsthesamefull,privatelinearaddressspace

Isolatesaddressspaces

Oneprocess

cant

interfere

with

anothers

memory

becausetheyoperateindifferentaddressspaces

differentsectionsofaddressspaceshavedifferentpermissions


23/61

Address Translation: Page TablesAddress Translation: Page Tables Apagetableisanarrayofpagetableentries(PTEs)that

mapsvirtualpagestophysicalpages.Here:8VPs

PerprocesskerneldatastructureinDRAM

Physicalmemory

(DRAM)Physicalpage

nullVP7

VP4

Valid

0

1

diskaddress

PTE0

PP0VP2

VP1

PP3

null Virtualmemory

(disk)

0

1

0

0

Memoryresident

pagetable

1PTE

7 VP1

VP2

VP3

VP4

VP6

VP7


24/61

Address Translation With a Page TableAddress Translation With a Page Table

VirtualaddressPagetable

(PTBR)

PagetablePa etableaddressValid Physicalpagenumber(PPN)forprocess

Validbit=0:pagenotinmemory

Physicaladdress


25/61

SomenotesadoptedfromBryantandOHallaron


26/61

Control FlowControl Flow Computers

do

Only

One

Thing

Fromstartuptoshutdown,aCPUsimplyreadsand

executes(interprets)asequenceofinstructions

Thissequenceisthesystemsphysicalcontrolflow(orflowofcontrol).

Physical control f low

1

inst2inst3

me

instn


27/61

Altering the Control FlowAltering the Control Flow Up

to

Now:

two

mechanisms

for

changing

control

flow:

Jumpsandbranches

Callandreturnusingthestackdiscipline.

Bothreacttochangesinprogramstate.

nsu c en orause u sys em

DifficultfortheCPUtoreacttochangesinsystemstate.

.

Instructiondividesbyzero

Userhitsctlcatthekeyboard

ystem

t mer

exp res Systemneedsmechanismsforexceptionalcontrolflow


28/61

Exceptional Control FlowExceptional Control Flow Mechanisms

for

exceptional

control

flow

exists

at

all

levels

ofacomputersystem.

LowlevelMechanism

exceptions

c ange ncon ro ow nresponse oasys emeven .e., c ange n

systemstate)

Combinationof

hardware

and

OS

software

HigherLevelMechanisms

Processcontextswitch

Signals Nonlocaljumps(setjmp/longjmp)

Imp emente ye t er:

OS

software

(context

switch

and

signals). Clanguageruntimelibrary:nonlocaljumps.

S fS f


29/61

System context for exceptionsSystem context for exceptions

ProcessorInterrupt Serial port Parallel portKeyboard

Local/IO Bus

con ro er con ro er con ro er con ro er

MemoryNetwork

ada ter IDE disk

controller

Video

adapterSCSI

controller

Display Network

SCSI bus

disk

disk CDROM


30/61

ExceptionsExceptions An

exception

is

atransfer

of

control

to

the

OS

in

response

tosomeevent (i.e.,changeinprocessorstate)

UserProcess OS

exception

exceptionprocessingevent I_current

I_nextyexcep on an er

returntoI_current returntoI_next abort

Examples:

divby0,arithmeticoverflow,pagefault,I/Orequest

completes,Ctrl

C

II


31/61

Interrupt VectorsInterrupt Vectors Eachtypeofeventhas

auniqueexceptionExceptionnumberk

Indexintojumptablecode forexception handler 0

num ers

a. .a.,interrupt

vector)

interrupt

vector

0

code for

exception handler 1

ump a een ry

pointstoafunction2 ...

n-1

exception handler 2

... .

Handlerkiscalledeachcode forexception handler n-1

occurs.


32/61

Asynchronous Exceptions (Interrupts)Asynchronous Exceptions (Interrupts) Caused

by

events

external

to

the

processor

Indicatedbysettingtheprocessorsinterruptpin

handlerreturns

to

next

instruction.

I/Ointerrupts

hittin ctlc at the ke board

arrivalofapacketfromanetwork

arrivalofadatasectorfromadisk

Hardreset

interrupt

hittingtheresetbutton

Softresetinterrupt

hittingctl

alt

delete

on

aPC

A T l H d SA T l H d S


33/61

A Typical Hardware SystemA Typical Hardware System

register file

CPU chip

ALU

system bus memory bus

main

memoryI/O

bridgebus interface

I/O bus Ex ansion slots for

disk

controller

graphics

adapter

USB

controller

other devices suchas network adapters.

mousekeyboard monitor

disk

R d D k S SR d D k S S


34/61

Reading a Disk Sector: StepReading a Disk Sector: Step 11

register file

CPU chipCPU initiates a disk read by writing a

command, logical block number, and

destination memory address to a port

(address) associated with disk controller.

main

memorybus interface

I/O bus

diskcontroller

graphicsadapter

USBcontroller


disk



35/61

Reading a Disk Sector: Step 2Reading a Disk Sector: Step 2

register file

CPU chipDisk controller reads the sector and

performs a direct memory access (DMA)

transfer into main memory.

main

memorybus interface

I/O bus

diskcontroller

graphicsadapter

USBcontroller


disk



36/61

Reading a Disk Sector: StepReading a Disk Sector: Step 33

register file

CPU chipWhen the DMA transfer completes, the

disk controller notifies the CPU with an

interrupt (i.e., asserts a special interrupt

pin on the CPU)

main

memorybus interface

I/O bus

diskcontroller

graphicsadapter

USBcontroller


disk

S h E iS h E i


37/61

Synchronous ExceptionsSynchronous Exceptions Caused

by

events

that

occur

as

aresult

of

executing

an

instruction:

Traps

Intentional

Exam les:s stemcalls break ointtra s s ecialinstructions

Returnscontroltonextinstruction

Faults Unintentionalbutpossiblyrecoverable

Examples:pagefaults(recoverable),protectionfaults(unrecoverable).

Eitherreexecutesfaulting(current)instructionoraborts.

Aborts unintentionalandunrecoverable

xamp es:par yerror,mac nec e c .

Abortscurrentprogram

T E lT E l


38/61

Trap ExampleTrap Example Opening

aFile

Usercallsopen(filename,options)0804d070 :

. . .804d082: cd 80 int $0x80

Functionopenexecutessystemcallinstructionint

. . .

OSmustfindorcreatefile,getitreadyforreadingor

writing

eturns

nteger

e

escr ptorUser Process OS

Open file

return

pop

F lt E l #1F lt E l #1


39/61

Fault Example #1Fault Example #1 Memory

Reference

Userwritestomemorylocation

That

portion

(page)

of

users

memory

is

currently

on

disk Pa ehandlermustload a einto h sicalmemor

Returnstofaultinginstruction

Successfulon

second

tr

int a[1000];

main ()

{

User Process OS

a[500] = 13;

}

page fault

Create page and loadevent

movl

into memoryreturn

F lt E l #2F lt E l #2


40/61

Fault Example #2Fault Example #2 Memory

Reference

Userwritestomemorylocation int a[1000];

Address

is

not

valid Pa ehandlerdetectsinvalidaddress

main ()

{ a[5000] = 13;

}

SendsSIGSEGsignaltouserprocess

User

rocessexits

with

se mentation

fault

User Process OS

page faulteventmovl

e ec nva a ress

Signal process

E ti T bl IA32 (E t)E ti T bl IA32 (E t)


41/61

Exception Table IA32 (Excerpt)Exception Table IA32 (Excerpt)

Exception Number Description ExceptionClass0 Divideerror Fault

13 Generalprotectionfault Fault

18 Machinecheck Abort

32

127 OS

defined Interrupt

or

trap128(0x80) Systemcall Trap

129255 OSdefined Interruptortrap

Checkpp.183:


42/61

Somenotes

adopted

from

Bryant

and

OHallaron

E l C P gE l C P g


43/61

Example C ProgramExample C Program

main.c swap.c

int buf[2] = {1, 2};

int main()

extern int buf[];

static int *bufp0 = &buf[0];*

swap();return 0;

}

void swap()

{ n emp;

bufp1 = &buf[1];temp = *bufp0;*bufp0 = *bufp1;*bufp1 = temp;

}

Static LinkingStatic Linking


44/61

Static LinkingStatic Linking Programs

are

translated

and

linked

using

acompiler

driver:

unix>gcc O2 g opmain.cswap.c

unix>./p

main.c swap.c Sourcefiles

(cpp,cc1,as) (cpp,cc1,as)

Separatelycompiled

Linker(ld)

. .relocatableobjectfiles

pFullylinkedexecutableobjectfile(containscodeanddataforallfunctionsdefinedinmain.candswap.c

Why Linkers? Modularity!Why Linkers? Modularity!


45/61

Why Linkers? Modularity!Why Linkers? Modularity! Program

can

be

written

as

acollection

of

smaller

sourcefiles,ratherthanonemonolithicmass.

thislater)

. .,

,

Why Linkers? Efficiency!Why Linkers? Efficiency!


46/61

Why Linkers? Efficiency!Why Linkers? Efficiency! Time:

Separate

Compilation

Changeonesourcefile,compile,andthenrelink.

No

need

to

recompile

other

source

files.

Space:Libraries

...

Yetexecutablefilesandrunningmemoryimagescontain

What Do Linkers Do?What Do Linkers Do?


47/61

What Do Linkers Do?What Do Linkers Do? Step

1:

Symbol

resolution

Programsdefineandreferencesymbols(variablesand

functions):

voidswap(){} /*definesymbolswap*/

swap(); /*referencesymbolswap*/

int *xp =&x; /*definexp,referencex*/

ym o e n t onsarestore ycomp er nsym otable.

Eachentryincludesname,type,size,andlocationofsymbol.

onesymboldefinition.

What Do Linkers Do? (cont )What Do Linkers Do? (cont )


48/61

What Do Linkers Do? (cont.)What Do Linkers Do? (cont.) Step

2:

Relocation

Mergesseparatecodeanddatasectionsintosingle

sections

Relocatessymbolsfromtheirrelativelocationsinthe.o

files

to

their

final

absolute

memory

locations

in

the

executable.

Updatesall

references

to

these

symbols

to

reflect

their

newpositions.

Three Kinds of Object Files (Modules)Three Kinds of Object Files (Modules)


49/61

Three Kinds of Object Files (Modules)Three Kinds of Object Files (Modules) Relocatable object

file

(.o

file)

Containscodeanddatainaformthatcanbecombinedwith

o erre oca a e o ec es o ormexecu a eo ec e.

Each.o

file

is

produced

from

exactly

one

source

(.c)

file

Containscodeanddatainaformthatcanbecopieddirectlyinto

memoryand

then

executed.

Sharedobjectfile(.sofile)

Specialtypeofrelocatable objectfilethatcanbeloadedinto

memoryand

linked

dynamically,

at

either

load

time

or

run

time.

CalledDynamicLinkLibraries(DLLs)byWindows

Packaging Commonly Used Functions`Packaging Commonly Used Functions`


50/61

Packaging Commonly Used FunctionsPackaging Commonly Used Functions How

to

package

functions

commonly

used

by

programmers?

Math,I/O,memorymanagement,stringmanipulation,

etc.

Awkward,giventhelinkerframeworksofar:

O tion1:

Put

all

functions

into

asin le

source

file

Programmerslinkbigobjectfileintotheirprograms

Spaceandtimeinefficient

Option2:

Put

each

function

in

aseparate

source

file

Programmersexplicitlylinkappropriatebinariesintotheir

programs

More

efficient,

but

burdensome

on

the

programmer

Solution: Static LibrariesSolution: Static Libraries


51/61

Solution: Static LibrariesSolution: Static Libraries Static

libraries

(.a

archive

files)

Concatenaterelatedrelocatableobjectfilesintoasingle

filewithanindex(calledanarchive).

Enhancelinkersothatittriestoresolveunresolved

external

references

by

looking

for

the

symbols

in

one

or

morearchives.

Ifan

archive

member

file

resolves

reference,

link

into

executable.

Creating Static LibrariesCreating Static Libraries


52/61

Creating Static LibrariesCreating Static Libraries

atoi.c printf.c random.c

Translator Translator ... Translator

atoi.o printf.o random.o

libc.a

Archiver (ar).

atoi.o printf.o random.o

C standard librar

Archiver allowsincrementalupdates

Recompilefunctionthatchangesandreplace.ofile

inarchive.

Commonly Used LibrariesCommonly Used Libraries


53/61

Commonly Used LibrariesCommonly Used Libraries libc.a (the

Cstandard

library)

8MBarchiveof900objectfiles.

, , , , ,

numbers,integermath

libm.a (theCmathlibrary)

1MBarc iveo 226o ject i es.

floatingpointmath(sin,cos,tan,log,exp,sqrt,)

% ar -t /usr/lib/libc.a | sort % ar -t /usr/lib/libm.a | sortfork.ofprintf.o

e_acos.oe_acosf.oe_acosh.o

fpu_control.o

fputc.ofreopen.ofscanf.o

e_acoshf.o

e_acoshl.oe_acosl.oe_asin.o

.fstab.o

_ .e_asinl.o

Linking with Static LibrariesLinking with Static Libraries


54/61

Linking with Static LibrariesLinking with Static Libraries

addvec.o multvec.o

main2.c vector.h Archiver(ar)

Translators

(cpp,cc1,as) libc.alibvector.a Staticlibraries

main2.o printf.oandanyother

modulescalledbyprintf.oaddvec.oRelocatableobjectfiles

p2Fullylinked

Using Static LibrariesUsing Static Libraries


55/61

U g S LU g S L Linkers

algorithm

for

resolving

external

references:

Scan.ofilesand.afilesinthecommandlineorder.

Duringthescan,keepalistofthecurrentunresolvedreferences.

Aseach

new

.o

or

.a

file,

obj,

is

encountered,

try

to

resolve

each

obj.

If

any

entries

in

the

unresolved

list

at

end

of

scan,

then

error. Problem:

Commandlineordermatters!

Moral:

put

libraries

at

the

end

of

the

command

line.

unix> gcc -L. libtest.o -lmine- . - .

libtest.o: In function `main':libtest.o(.text+0x4): undefined reference to `libfun'

Shared LibrariesShared Libraries


56/61

S LS L Static

libraries

have

the

following

disadvantages:

Duplicationinthestoredexecutables(everyfunction

needstdlibc)

Duplicationintherunningexecutables

Minorbugfixesofsystemlibrariesrequireeach

applicationtoexplicitlyrelink

ModernSolution:SharedLibraries

Objectfilesthatcontaincodeanddatathatareloaded

andlinked

into

an

application

dynamically,

at

either

loadtimeorruntime

Alsocalled:dynamiclinklibraries,DLLs,.sofiles

Shared Libraries (cont.)Shared Libraries (cont.)


57/61

S L ( )S L ( ) Dynamic

linking

can

occur

when

executable

is

first

loaded

andrun(loadtimelinking).

CommoncaseforLinux,handledautomaticallybythedynamic

linker(ld

linux.so).

. .

Dynamiclinkingcanalsooccurafterprogramhasbegun

runtime

linkin .

InUnix,thisisdonebycallstothedlopen()interface.

Highperformancewebservers.

Runtimelibraryinterpositioning

Sharedlibraryroutinescanbesharedbymultiple

.

Moreonthiswhenwelearnaboutvirtualmemory

Dynamic Linking at LoadDynamic Linking at Load--timetime


58/61

Dy L g LDy L g L

Translators

main2.c vector.h unix> gcc -shared -o libvector.so \addvec.c multvec.c

(cpp,cc1,as)

main2.o

libc.solibvector.so

RelocationandsymbolRelocatableob ect ile

Linker(ld)

p2Partiallylinkedexecutableobjectfile

libc.so

libvector.so

Codeanddata

(execve)

Dynamiclinker(ld-linux.so)Fullylinkedexecutable

inmemory

Refined View of MemoryRefined View of Memory


59/61

yy

Kernelvirtualmemory

Userstack

Memoryinvisibleto

usercode0xc0000000

(createdatruntime)%esp

(stack

pointer)

Memorymappedregionfor

sharedlibraries0x40000000

Runtimeheap

(createdbymalloc)

brk

Read/writesegment(.data,.bss)

Loadedfrom

the

Unused0

0x08048000

(.init,.text,.rodata)

file

Case Study: LibraryCase Study: Library InterpositioningInterpositioning


60/61

y yy y p gp g Library

interpositioning is

apowerful

linking

techniquethatallowsprogrammerstointercept

callstoarbitraryfunctions

Inter ositionin canoccurat:

compiletime

Whenthe

source

code

is

com iled

linktime

Whentherelocatable objectfilesarelinkedtoforman

executable

object

file load/runtime

Whenanexecutableobjectfileisloadedintomemory,

dynamicallylinked,andthenexecuted.

Some Interpositioning ApplicationsSome Interpositioning Applications


61/61

p g ppp g pp Security

Confinement(sandboxing)

Interposecallstolibc functions.

Behindthescenesencryption

Automaticallyencryptotherwiseunencryptednetwork

connections.

MonitoringandProfiling

Countnumberofcallstofunctions

Characterizecall

sites

and

arguments

to

functions

Malloc tracing

Detectingmemoryleaks

Generatingmalloc traces

Documents

Cs33 Os Basics