The Machine and the KernelMode, space, and context: the basics

Jeff ChaseDuke University

64 bytes: 3 waysp + 0x0

0x1f

0x0

0x1f

0x1f

0x0

char p[]char *p

int p[]int* p

p

char* p[]char** p

Pointers (addresses) are 8 bytes on a 64-bit machine.

Memory is “fungible”.

Endianness

A silly difference among machine architectures creates a need for byte swapping when unlike machines exchange data over a network.

Lilliput and Blefuscu are at war over which end of a soft-boiled egg to crack.

Gulliver’s Travel’s 1726

x86 is little-endian

chase$ cc -o heap heap.cchase$ ./heaphi!0x216968chase$

h=0x68i=0x69!=0x21

0

cbip

Little-endian: the lowest-numbered byte of a word (or longword or quadword) is the least significant.

Network messages

https://developers.google.com/protocol-buffers/docs/overview

Byte swapping: example

struct sockaddr_in socket_addr;

sock = socket(PF_INET, SOCK_STREAM, 0);

memset(&socket_addr, 0, sizeof socket_addr); socket_addr.sin_family = PF_INET; socket_addr.sin_port = htons(port); socket_addr.sin_addr.s_addr = htonl(INADDR_ANY);

if (bind(sock, (struct sockaddr *) &socket_addr, sizeof socket_addr) < 0) { perror("couldn't bind"); exit(1); } listen(sock, 10);

buggyserver.c

Heap: dynamic memory

Allocated heap blocks for structs or objects.

Align!

A contiguous chunk of memory obtained from

OS kernel.E.g., with Unix sbrk()

system call.

A runtime library obtains the block and manages it as a

“heap” for use by the programming language environment, to store

dynamic objects.

E.g., with Unix malloc and free library calls.

Heap manager policy

• The heap manager must find a suitable free block to return for each call to malloc().– No byte can be part of two simultaneously allocated heap

blocks! If any byte of memory is doubly allocated, programs will fail. We test for this!

• A heap manager has a policy algorithm to identify a suitable free block within the heap.– Last fit, first fit, best fit, worst fit

– Choose your favorite!

– Goals: be quick, and use memory efficiently

– Behavior depends on workload: pattern of malloc/free requests

• This is an old problem in computer science, and it occurs in many settings: variable partitioning.

Variable PartitioningVariable partitioning is the strategy of parking differently sized carsalong a street with no marked parking space dividers.

Wasted spaceexternal fragmentation

2

3

1

Fixed Partitioning

Wasted spaceinternal fragmentation

Time sharing vs. space sharing

time

space

Two common modes of resource

allocation. What kinds of resources do

these work for?

Operating Systems: The Classical View

data dataPrograms

run asindependent processes.

Protected system calls

...and upcalls (e.g., signals)

Protected OS kernel

mediates access to

shared resources.

Threads enter the kernel for

OS services.

Each process has a private

virtual address space and one

or more threads.

The kernel code and data are protected from untrusted processes.

0x0

0x7fffffff

Static data

Dynamic data(heap/BSS)

Text(code)

Stack

Reserved

0x0

0x7fffffff

Static data

Dynamic data(heap/BSS)

Text(code)

Stack

Reserved

“Classic Linux Address Space”

http://duartes.org/gustavo/blog/category/linux

N

Windows/IA32

Windows IA-32(Kernel)

Processes: A Closer Look

+ +user ID

process IDparent PIDsibling links

children

virtual address space process descriptor (PCB)

resources

thread

stack

Each process has a thread bound to the VAS.

The thread has a stack addressable through the

VAS.

The kernel can suspend/restart the thread wherever and whenever it

wants.

The OS maintains some state for each

process in the kernel’s internal

data structures: a file descriptor table, links to maintain the process tree, and a place to store the

exit status.

The address space is a private name space for a set of memory

segments used by the process.

The kernel must initialize the process

memory for the program to run.

A process can have multiple threads

volatile int counter = 0; int loops;

void *worker(void *arg) { int i; for (i = 0; i < loops; i++) {

counter++; } pthread_exit(NULL);}

int main(int argc, char *argv[]) { if (argc != 2) {

fprintf(stderr, "usage: threads <loops>\n"); exit(1);

} loops = atoi(argv[1]); pthread_t p1, p2; printf("Initial value : %d\n", counter); pthread_create(&p1, NULL, worker, NULL); pthread_create(&p2, NULL, worker, NULL); pthread_join(p1, NULL); pthread_join(p2, NULL); printf("Final value : %d\n", counter); return 0;}

data

Much more on this later!

Key Concepts for Classical OS

• kernel• The software component that controls the hardware directly, and

implements the core privileged OS functions.

• Modern hardware has features that allow the OS kernel to protect itself from untrusted user code.

• thread• An executing instruction path and its CPU register state.

• virtual address space• An execution context for thread(s) defining a name space for

executing instructions to address data and code.

• process• An execution of a program, consisting of a virtual address

space, one or more threads, and some OS kernel state.

The theater analogy

Threads

Address space

Program

script

virtual memory(stage)

[lpcox]

Running a program is like performing a play.

The sheep analogy

Thread

Code and data

Address space

CPU cores

Core #1 Core #2

The machine has a bank of CPU cores for threads to run on.

The OS allocates cores to threads.

Cores are hardware. They go where the driver tells them.

Switch drivers any time.

Threads drive cores

What was the point of that whole thing with the electric sheep actors?

• A process is a running program.

• A running program (a process) has at least one thread (“main”), but it may (optionally) create other threads.

• The threads execute the program (“perform the script”).

• The threads execute on the “stage” of the process virtual memory, with access to a private instance of the program’s code and data.

• A thread can access any virtual memory in its process, but is contained by the “fence” of the process virtual address space.

• Threads run on cores: a thread’s core executes instructions for it.

• Sometimes threads idle to wait for a free core, or for some event. Sometimes cores idle to wait for a ready thread to run.

• The operating system kernel shares/multiplexes the computer’s memory and cores among the virtual memories and threads.

Processes and threads

+ +…

virtual address space main thread

stack

Each process has a thread bound to the VAS, with

stacks (user and kernel).

If we say a process does something, we really mean

its thread does it.

The kernel can suspend/restart the thread wherever and whenever it

wants.

Each process has a virtual address space (VAS): a private name space for the virtual

memory it uses.

The VAS is both a “sandbox” and a

“lockbox”: it limits what the process can

see/do, and protects its data from others.

From now on, we suppose that a process could have

multiple threads.

We presume that they can all make system calls and

block independently.

other threads (optional)

STOP wait

A thread running in a process VAS

0

high

code library

your data

heap

registers

CPU

R0

Rn

PC

“memory”

x

x

your program

common runtime

stack

address space(virtual or physical)

e.g., a virtual memory for a process

SP

y

y

Thread context

• Each thread has a context (exactly one).– Context == values in the thread’s registers

– Including a (protected) identifier naming its VAS.

– And a pointer to thread’s stack in VAS/memory.

• Each core has a context (at least one).– Context == a register set that can hold values.

– The register set is baked into the hardware.

• A core can change “drivers”: context switch.– Save running thread’s register values into memory.

– Load new thread’s register values from memory.

– (Think of driver settings for the seat, mirrors, audio…)

– Enables time slicing or time sharing of machine.

registers

CPU core

R0

Rn

PC xSP y

Programs gone wild

intmain(){

while(1);}

Can you hear the fans blow?

How does the OS regain control of the core from this program?

How to “make” the process save its context and give some other process a chance to run?

How to “make” processes share machine resources fairly?

Timer interrupts, faults, etc.

• When processor core is running a user program, the user program/thread controls (“drives”) the core.

• The hardware has a timer device that interrupts the core after a given interval of time.

• Interrupt transfers control back to the OS kernel, which may switch the core to another thread, or resume.

• Other events also return control to the kernel.– Wild pointers

– Divide by zero

– Other program actions

– Page faults

Entry to the kernel

syscall trap/return fault/return

interrupt/return

The handler accesses the core register context to read the details of the exception (trap, fault, or interrupt). It may call other kernel routines.

Every entry to the kernel is the result of a trap, fault, or interrupt. The core switches to kernel mode and transfers control to a handler routine.

OS kernel code and data for system calls (files, process fork/exit/wait, pipes, binder IPC, low-level thread support, etc.)

and virtual memory management (page faults, etc.)

I/O completions timer ticks

registers

CPU core

R0

Rn

PC x

mode

CPU mode (a field in some status register) indicates whether a machine CPU (core) is running in a user program or in the protected kernel (protected mode).

Some instructions or register accesses are legal only when the CPU (core) is executing in kernel mode.

CPU mode transitions to kernel mode only on machine exception events (trap, fault, interrupt), which transfers control to a trusted handler routine registered with the machine at kernel boot time.

So only the kernel program chooses what code ever runs in the kernel mode (or so we hope and intend).

A kernel handler can read the user register values at the time of the event, and modify them arbitrarily before (optionally) returning to user mode.

CPU mode: User and Kernel

U/K

synchronouscaused by an

instruction

asynchronouscaused by some other

event

intentionalhappens every time

unintentionalcontributing factors

trap: system callopen, close, read,

write, fork, exec, exit, wait, kill, etc.

faultinvalid or protected

address or opcode, page fault, overflow, etc.

interruptcaused by an external

event: I/O op completed, clock tick, power fail, etc.

“software interrupt” software requests an

interrupt to be delivered at a later time

Exceptions: trap, fault, interrupt

Kernel Stacks and Trap/Fault Handling

data

Threads execute user

code on a user stack in the user virtual

memory in the process virtual address space.

Each thread has a second kernel stack in kernel

space (VM accessible only in

kernel mode).

stack

stack

stack

stack

System calls and faults run

in kernel mode on a

kernel stack.

syscall dispatch

table

Kernel code running in P’s

process context has access to

P’s virtual memory.

The syscall handler makes an indirect call through the system call dispatch table to the handler registered for the specific system call.

Virtual resource sharing

time

space

Understand that the OS kernel implements resource allocation (memory, CPU,…) by manipulating name spaces and contexts visible to user code.

The kernel retains control of user contexts and address spaces via the machine’s limited direct execution model, based on protected mode and exceptions.

“Limited direct execution”

user mode

kernel mode

kernel “top half”kernel “bottom half” (interrupt handlers)

syscall trap

u-start u-return u-start

fault

u-return

fault

clock interrupt

interruptreturn

Kernel handler manipulates CPU register context to return

to selected user context.

Any kind of machine exception transfers control to a registered (trusted) kernel handler running in a protected CPU mode.

boot

Example: Syscall traps

• Programs in C, C++, etc. invoke system calls by linking to a standard library written in assembly.– The library defines a stub or wrapper routine for each syscall.

– Stub executes a special trap instruction (e.g., chmk or callsys or syscall instruction) to change mode to kernel.

– Syscall arguments/results are passed in registers (or user stack).

– OS defines Application Binary Interface (ABI).

read() in Unix libc.a Alpha library (executes in user mode):

#define SYSCALL_READ 27 # op ID for a read system call

move arg0…argn, a0…an # syscall args in registers A0..ANmove SYSCALL_READ, v0 # syscall dispatch index in V0callsys # kernel trapmove r1, _errno # errno = return statusreturn

Alpha CPU ISA (defunct)

Linux x64 syscall conventions

MacOS x86-64 syscall examplesection .datahello_world     db      "Hello World!", 0x0a section .textglobal start start:mov rax, 0x2000004      ; System call write = 4mov rdi, 1              ; Write to standard out = 1mov rsi, hello_world    ; The address of hello_world stringmov rdx, 14             ; The size to writesyscall                 ; Invoke the kernelmov rax, 0x2000001      ; System call number for exit = 1mov rdi, 0              ; Exit success = 0syscall                 ; Invoke the kernel

http://thexploit.com/secdev/mac-os-x-64-bit-assembly-system-calls/

Illustration only: this program writes “Hello World!” to standard output.

A thread running in a process VAS

0

high

code library

your data

heap

registers

CPU

R0

Rn

PC

“memory”

x

x

your program

common runtime

stack

address space(virtual or physical)

e.g., a virtual memory for a process

SP

y

y

Messing with the context#include <ucontext.h>

int count = 0;ucontext_t context;

int main(){ int i = 0; getcontext(&context);

count += 1; i += 1; sleep(2); printf(”…", count, i);

setcontext(&context);}

ucontextStandard C library routines to:

Save current register context to a block of memory (getcontext from core)

Load/restore current register context from a block of memory (setcontext)

Also: makecontext, swapcontext

Details of the saved context (ucontext_t structure) are machine-dependent.

Messing with the context (2)#include <ucontext.h>

int count = 0;ucontext_t context;

int main(){ int i = 0; getcontext(&context);

count += 1; i += 1; sleep(1); printf(”…", count, i);

setcontext(&context);}

Loading the saved context transfers control to this block

of code. (Why?)

What about the stack?

Save core context to memory

Load core context from memory

Messing with the context (3)#include <ucontext.h>

int count = 0;ucontext_t context;

int main(){ int i = 0; getcontext(&context);

count += 1; i += 1; sleep(1); printf(”…", count, i);

setcontext(&context);}

chase$ cc -o context0 context0.c< warnings: ucontext deprecated on MacOS >chase$ ./context0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 …

Reading behind the C

Disassembled code:

movl 0x0000017a(%rip),%ecxaddl $0x00000001,%ecxmovl %ecx,0x0000016e(%rip)

movl 0xfc(%rbp),%ecxaddl $0x00000001,%ecxmovl %ecx,0xfc(%rbp)

%rip and %rbp are set “right”, then these references “work”.

count += 1;i += 1;

On MacOS:chase$ man otoolchase$ otool –vt context0 …

On this machine, with this cc:

Static global _count is addressed relative to the location of the code itself, as given by the PC register [%rip is instruction pointer register]

Local variable i is addressed as an offset from stack frame.[%rbp is stack frame base pointer]

Messing with the context (4)#include <ucontext.h>

int count = 0;ucontext_t context;

int main(){ int i = 0; getcontext(&context);

count += 1; i += 1; sleep(1); printf(”…", count, i);

setcontext(&context);}

chase$ cc –O2 -o context0 context0.c< warnings: ucontext deprecated on MacOS >chase$ ./context0 1 1 2 1 3 1 4 1 5 1 6 1 7 1 …

What happened?

The point of ucontext

• The system can use ucontext routines to:– “Freeze” at a point in time of the execution

– Restart execution from a frozen moment in time

– Execution continues where it left off…if the memory state is right.

• The system can implement multiple independent threads of execution within the same address space.– Create a context for a new thread with makecontext.

– Modify saved contexts at will.

– Context switch with swapcontext: transfer a core from one thread to another (“change drivers”)

– Much more to this picture: need per-thread stacks, kernel support, suspend/sleep, controlled ordering, etc.

Two threads: closer look

0

high

code library

data

registers

CPU(core)

R0

Rn

PC

x

x

program

common runtime

stack

address space

SP

y

y stack

runningthread

“on deck” and ready to run

Thread context switch

0

high

code library

data

registers

CPU(core)

R0

Rn

PC

x

x

program

common runtime

stack

address space

SP

y

y stack

1. save registers

2. load registers

switch in

switch out

A metaphor: context/switching

Page links and back button navigate a

“stack” of pages in each tab.

Each tab has its own stack.One tab is active at any given time.

You create/destroy tabs as needed.You switch between tabs at your whim.

Similarly, each thread has a separate stack.The OS switches between threads at its whim.

One thread is active per CPU core at any given time.

1

2

3

time

Messing with the context (5)

#include <ucontext.h>

int count = 0;ucontext_t context;

int main(){ int i = 0; getcontext(&context); count += 1; i += 1; sleep(1); printf(”…", count, i);

setcontext(&context);}

What does this do?

Thread/process states and transitions

running

readyblocked

Scheduler governs these transitions.

wait, STOP, read, write, listen, receive, etc.

sleep

STOP wait

wakeup

Sleep and wakeup are internal primitives. Wakeup adds a thread to the scheduler’s ready pool: a set of

threads in the ready state.

yield

“requesting a car”

“driving a car”

“waiting for someplace to go”

dispatch

BLOCK MAPS AND PAGE TABLES

Blocks are contiguous

The storage in a heap block is contiguous in the Virtual Address Space.The term block always refers to a contiguous sequence of bytes suitable for base+offset addressing.

C and other PL environments require this. E.g., C compiler determines the offsets for named fields in a struct and “bakes” them into the code.

This requirement complicates the heap manager because the heap blocks may be different sizes.

Block maps

map

Large data objects may be mapped so they don’t have to

be stored contiguously in machine memory.

(e.g., files, segments)

Idea: use a level of indirection through a map to assemble a

storage object from “scraps” of storage in different locations.

The “scraps” can be fixed-size slots: that makes allocation easy because the slots are

interchangeable (fixed partitioning).

Example: page tables that implement a VAS.

x64, x86-64, AMD64: VM Layout

Source:

System V Application Binary Interface AMD64

Architecture Processor Supplement 2005

VM page map

Indirection

Fixed Partitioning

Wasted spaceinternal fragmentation

Names and maps

• Block maps and other indexed maps are common structure to implement “machine” name spaces:– sequences of logical blocks, e.g., virtual address spaces, files

– process IDs, etc.

– For sparse block spaces we may use a tree hierarchy of block maps (e.g., inode maps or 2-level page tables, later).

– Storage system software is full of these maps.

• Symbolic name spaces use different kinds of maps.– They are sparse and require matching more expensive.

– Property list, key/value hash table

– Trees of maps create nested namespaces, e.g., the file tree.

EXTRA SLIDESI hope we get to here

The Kernel• Today, all “real” operating systems have protected kernels.

The kernel resides in a well-known file: the “machine” automatically loads it into memory (boots) on power-on/reset.

Our “kernel” is called the executive in some systems (e.g., Windows).

• The kernel is (mostly) a library of service procedures shared by all user programs, but the kernel is protected:

User code cannot access internal kernel data structures directly.

User code can invoke the kernel only at well-defined entry points (system calls).

• Kernel code is “just like” user code, but the kernel is privileged:

The kernel has direct access to all hardware functions, and defines the handler entry points for interrupts and exceptions.

Protecting Entry to the Kernel

Protected events and kernel mode are the architectural foundations of kernel-based OS (Unix, Windows, etc).– The machine defines a small set of exceptional event types.

– The machine defines what conditions raise each event.

– The kernel installs handlers for each event at boot time.

e.g., a table in kernel memory read by the machine

The machine transitions to kernel mode only on an exceptional event.

The kernel defines the event handlers.

Therefore the kernel chooses what code will execute in kernel mode, and when.

user

kernel

interrupt orfault

trap/returninterrupt orfault

The Role of Events

• A CPU event (an interrupt or exception, i.e., a trap or fault) is an “unnatural” change in control flow.

• Like a procedure call, an event changes the PC register.

• Also changes mode or context (current stack), or both.

• Events do not change the current space!

• On boot, the kernel defines a handler routine for each event type.• The machine defines the event types.

• Event handlers execute in kernel mode.

• Every kernel entry results from an event.• Enter at the handler for the event.

control flow

event handler (e.g., ISR: Interrupt Service Routine)

exception.cc

In some sense, the whole kernel is a “big event handler.”

Examples

• Illegal operation– Reserved opcode, divide-by-zero, illegal access

– That’s a fault! Kernel generates a signal, e.g., to kill process or invoke PL exception handlers.

• Page fault– Fetch and install page, maybe block process

– Nothing illegal about it: “transparent” to faulting process

• I/O completion, arriving input, clock ticks.– These external events are interrupts.

– Include power fail etc.

– Kernel services interrupt in handler.

– May wakeup blocked processes, but no blocking.

Faults

• Faults are similar to system calls in some respects:

– Faults occur as a result of a process executing an instruction.

• Fault handlers execute on the process kernel stack; the fault handler may block (sleep) in the kernel.

– The completed fault handler may return to the faulted context.

• But faults are different from syscall traps in other respects:

– Syscalls are deliberate, but faults are “accidents”.

• divide-by-zero, dereference invalid pointer, memory page fault

– Not every execution of the faulting instruction results in a fault.

• may depend on memory state or register contents

Note: Something Wild

• The “Something Wild” example that follows was an earlier version of “Messing with the context”.

• It was not discussed in class.

• “Messing with the context” simplifies the example, but keeps all the essential info.

• “Something Wild” brings it just a little closer to coroutines a context switch from one thread to another.

Something wild (1)

#include <ucontext.h>Int count = 0;int set = 0;ucontext_t contexts[2];

void proc() { int i = 0; if (!set) { getcontext(&contexts[count]); } printf(…, count, i); count += 1; i += 1; if (set) { setcontext(&contexts[count&0x1]); }}

time

int main() { set = 0; proc(); proc(); set = 1; proc();}

Something wild (2)

#include <ucontext.h>ucontext_t contexts[2];

void proc(){ int i = 0; getcontext(&contexts[count]); printf(”…", count, i); count += 1; i += 1;}

time

int main() { set=0; proc(); proc(); …}

Something wild (3)

#include <ucontext.h>ucontext_t contexts[2];

void proc() { int i = 0; printf(”…", count, i); count += 1; i += 1; sleep(1); setcontext(&contexts[count&0x1]);}

time

int main() { … set=1; proc();}

Something wild (4)

void proc() { int i = 0; printf(”…", count, i); count += 1; i += 1; sleep(1); setcontext(…);}

time

Switch to the other saved register context. Alternate “even” and

“odd” contexts.

We have a pair of register contexts that

were saved at this point in the code. If we load either of the

saved contexts, it will transfer control to this block of code. (Why?)

What about the stack?

Lather, rinse, repeat.What will it print? The count is a

global variable…but what about i?

void proc() { int i = 0; printf("%4d %4d\n", count, i); count += 1; i += 1; sleep(1); setcontext(…);}

Something wild (5)

time

What does this do?