Multiprocessor Initialization

An introduction to the use of Interprocessor Interrupts

A traditional MP system

CPU0

CPU1 Main memory

system bus

Core 2 Duo processor

Dual-Core Technology

CPU0

CPU1

Main memory

system bus

Shared level-2 cache

Multi-Core TechnologyCore 2 Quad processor

CPU0

CPU1

Main memory

system bus

Shared level-2 cache

CPU2

CPU3

Shared level-2 cache

CPU has its own Local-APIC

CPU

processor’s application registers EAX, EBX, …, EIP, EFLAGS

processor’s system registers CR0, CR2, CR3, …, IDTR, GDTR, TR

processor’s Local-APIC registersLocal-ID, IRR, ISR, EOI, LVT0, LVT1, …, ICR, TCFG

processor’s Execution Engine

The Local-APIC ID register

reservedAPIC

ID

31 24 0

Memory-Mapped Register-Address: 0xFEE00020

This register is initially zero, but its APIC ID Field (8-bits) is programmed by the BIOS during system startup with a unique processor identification-Number, which subsequently is used when specifying the processor as arecipient of inter-processor interrupts.

The Local-APIC EOI register

write-only register

31 0

Memory-Mapped Register-Address: 0xFEE000B0

This write-only register is used by Interrupt Service Routines to issue an‘End-Of-Interrupt’ command to the Local-APIC. Any value written to thisregister will be interpreted by the Local-APIC as an EOI command. Thevalue stored in this register is initially zero (and it will remain unchanged).

The Spurious Interrupt register

reserved spuriousvector

31 7 0

Memory-Mapped Register-Address: 0xFEE000F0

This register is used to Enable/Disable the functioning of the Local-APIC,and when enabled, to specify the interrupt-vector number to be deliveredto the processor in case the Local-APIC generates a ‘spurious’ interrupt.(In some processor-models, the vector’s lowest 4-bits are hardwired 1s.)

EN

8

Local-APIC is Enabled (1=yes, 0=no)

Interrupt Command Register

• Each processor’s Local-APIC unit has a 64-bit Interrupt Command Register

• It can be programmed by system software to transmit messages to one, or to several, of the other processors in the system

• Each processor has a unique identification number in its APIC Local-ID Register that can be used for directing messages to it

ICR (upper 32-bits)

reservedDestination

field

31 24 0

Memory-Mapped Register-Address: 0xFEE00310

The Destination Field (8-bits) can be used to specify whichprocessor (or group of processors) will receive the message

ICR (lower 32-bits)

Vectorfield

31 19 18 07

Destination Shorthand 00 = no shorthand 01 = only to self 10 = all including self 11 = all excluding self

R/O

10 8

Delivery Mode 000 = Fixed 001 = Lowest Priority 010 = SMI 011 = (reserved) 100 = NMI 101 = INIT 110 = Start Up 111 = (reserved)

Trigger Mode 0 = Edge 1 = Level

15

Level 0 = De-assert 1 = Assert

Destination Mode 0 = Physical 1 = Logical

12

Delivery Status 0 = Idle 1 = Pending Memory-Mapped Register-Address: 0xFEE00300

MP initialization protocol

• Set a shared processor-counter equal to 1• Step 1: issue an ‘INIT’ IPI to all-except-self• Delay for 10 milliseconds• Step 2: issue ‘Startup’ IPI to all-except-self• Delay for 200 microseconds• Step 3: issue ‘Startup’ IPI to all-except-self• Delay for 200 microseconds• Check the value of the processor-counter

Issue an ‘INIT’ IPI

# address Local-APIC via register FSmov $sel_fs, %axmov %ax, %fs# broadcast ‘INIT’ IPI to ‘all-except-

self’mov $0x000C4500, %eaxmov %eax, %fs:0xFEE00300)

.B0: btl $12, %fs:(0xFEE00300)jc .B0

Issue a ‘Startup’ IPI

# broadcast ‘Startup’ IPI to all-except-self

# using vector 0x11 to specify entry-point

# at real memory-address 0x00011000

mov $0x000C4611, %eax

mov %eax, %fs:(0xFEE00300)

.B1: btl $12, %fs:(0xFEE00300)

jc .B1

Timing delays

• Intel’s MP Initialization Protocol specifies the use of some timing-delays:– 10 milliseconds ( = 10,000 microseconds)– 200 microseconds

• We can use the 8254 Timer’s Channel 2 for implementing these timed delays, by programming it for ‘one-shot’ countdown mode, then polling bit #5 at i/o port 0x61

Mathematical examples

EXAMPLE 2Delaying for 200-microseconds means delaying 1/5000-th of a second (because 5000 times 200 microseconds = one-million microseconds)

EXAMPLE 1 Delaying for 10-milliseconds means delaying for 1/100-th of a second (because 100 times 10-milliseconds = one-thousand milliseconds)

GENERAL PRINCIPLEDelaying for x–microseconds means delaying for 1000000/x seconds (because 1000000/x times x-microseconds = one-million microseconds)

Mathematical theory

RECALL: Clock-Frequency-in-Seconds = 1193182 HertzALSO: One second equals one-million microseconds

PROBLEM: Given the desired delay-time in microseconds, express the desired delay-time in clock-frequency pulses and program that number into the PIT’s Latch-Register

Delay-in-Clock-Pulses = Delay-in-Microseconds * Pulses-Per-Microsecond

Pulses-Per-Microsecond = Pulses-Per-Second / Microseconds-Per-Second

APPLYING DIMENSIONAL ANALYSIS

CONCLUSION

For a desired time-delay of x microseconds, the number of clock-pulsesmay be computed as x * (1193182 /1000000) = (1193182 * x) / 1000000as dividing by a fraction amounts to multiplying by that fraction’s reciprocal

Delaying for EAX microseconds

# We compute the value for the 8254 Timer’s Channel-2 Latch-register# Delaying for EAX microseconds means that Latch-register’s value is # a certain fraction of one full second’s worth of input-pulses:# fraction = (EAX microseconds)/(one-million microseconds-per-second) # # Thus the latch-value should be: fraction*(1193182 pulses-per-second)# which we can compute by doing a multiplication followed by a division #

mov %eax, %ecx # copy the delay to ECX

mov $1193182, %eax # setup input-frequency in EAXmul %ecx # multiplied by microsecondsmov $1000000, %ecx # setup one-million as a divisordiv %ecx # so quotient will be Latch-value

# Quotient in register AX should be written to the timer’s Latch Register

Intel’s MP terminology

• When an MP system starts up, one of the CPUs will be selected to handle the ‘boot’ procedures, while the other CPUs ‘sleep’

• The BSP is this BootStrap Processor, and every other processor is known as an AP (i.e., a so-called ‘Application Processor’)

BSP AP AP AP

‘parallel computing’ principles

• When it’s awakened, each processor will need its own private stack-area, so it can handle any interrupts or procedure-calls without modifying an area in memory which another processor is also using

• And whenever two or more processors do share ‘write-access’ to any memory area, then those accesses must ‘serialized’

‘atomic’ memory-access

• Shared variables must not be modified by more than one processor at a time (‘atomic’ access)

• The x86 cpu’s ‘lock’ prefix helps enforce this• Example: every processor adds 1 to a counter

lockincl (counter)

• Some instructions have ‘atomic’ access built in • Example: all processors needs private stacks

mov 0x1000, %axxadd (new_SS), %axmov %ax, %ss

ROM-BIOS isn’t ‘reentrant’

• The video service-functions in ROM-BIOS often used to display a message-string at the current cursor-location (and afterward advance the cursor) modify global storage locations (as well as i/o ports), and hence must be called by one processor at a time

• A shared memory-variable (called ‘mutex’) is used to enforce this mutual exclusion

Implementing a ‘spinlock’

# Here is a ‘global’ variable, which all of the processors can modifymutex: .word 1 # initial value for variable is 1

# Here is a ‘prologue’ and ‘epilog’ for using this variable to enforce# ‘mutually exclusive access’ to a section of ‘non-reentrant’ code

spin: btw $0, mutex # test bit #0 to see if mutex is freejnc spin # spin if the mutex is not available

lock # else request exclusive bus-access btrw $0, mutex # and try to grab mutex ownershipjnc spin # unsuccessful? then try again

< CRITICAL SECTION OF ‘NON-REENTRANT’ CODE>

btsw $0, mutex # release the mutex when finished

Demo: ‘mphello.s’

• Each CPU needs to access its Local-APIC

• The BSP (“Boot-Strap Processor”) wakes up other processors by broadcasting the ‘INIT-SIPI-SIPI’ message-sequence

• Each AP (“Application Processor”) starts executing at a 4K page-boundary -- and needs its own private stack-area

• Shared variables require ‘atomic’ access

Demo’s organization

MAIN: # the BSP will execute these callscall allow_4GB_accesscall display_APIC_LocalIDcall broadcast_AP_starupcall delay_until_APs_halt

initAP: # each AP will execute these callscall allow_4GB_accesscall display_APIC_LocalID

In-class exercise #1

• Add a call to this procedure by each of the processors, but do it without using a ‘lock’ prefix (and outside mutex-protected code)

• Then let the BSP print the value of ‘total’

total: .word 0 # include this ‘shared’ global-variable

add_one_thousand: # let each processor call this subroutinemov $1000, %cx

nxadd: addw $1, totalloop nxaddret

Binary-to-Decimal

• Recall algorithm for converting numbers to decimal digit-strings (for console display)

num2dec: # converts value in register AX to a decimal string at DS:DImov $10, %bx # setup the number-base in BXxor %cx, %cx # setup remainder-count in CX

nxdiv: xor %dx, %dx # extend AX to a doubleworddiv %bx # divide the doubleword by tenpush %dx # save remainder on the stackinc %cx # and count this remainderor %ax, %ax # was the quotient zero yet?jnz nxdiv # no, generate another digit

nxdgt: pop %dx # recover saved remainderadd $’0’, %dl # convert remainder to ASCIImov %dl, (%di) # store numeral in output-bufferinc %di # and advance buffer-pointerloop nxdgt # again for other remainders

In-class exercise #2

• Using a Core-2 Quad processor we might expect the value of ‘total’ would be 4000

• But see if that’s what actually happens!

• Without the ‘lock’ prefix, the four CPUs may all try to increment ‘total’ at once, resulting in a logically incorrect total

• So fix this problem (by using a ‘lock’ prefix ahead of the ‘addw $1, total’ instruction)

Do you need a ‘barrier’?

• You can use a software construct, known as a ‘barrier’, to stop CPUs from entering a block of code until a prescribed number of them are all ready to enter it together (i.e., simultaneously)

• This may be helpful with the in-class exercises

arrived: .word 0 # allocate a shared global variable

barrier: lock # acquire exclusive bus-access incw arrived # each cpu adds 1 to the variable

await: cmpw $4, arrived # are four cpus ready to proceed?jb await # no, wait for others to arrive herecall add_one_thousand # then proceed together