7: Distributed Systems and Concurrency fileA “node” can be a computer of its own, a core on a given processor, or even one of several processes on a single-core machine. In any

7: Distributed Systems and Concurrency

Distributed System:

I several coputational entities (“nodes”) with their own local memory,I these entities communicate with each other (e.g., message

passing, or shared memory),I and the system presents itself as a homogenous system to the

user (or attempts to do so).

A “node” can be a computer of its own, a core on a given processor, oreven one of several processes on a single-core machine.In any case will refer to the program running on a node as a “process”.

This section deals with the core ideas and obstacles for implementingconcurrent and distributed systems, and gives some very basic hintshow these are implemented in Ada.

–352– S. Lucks Security Engineering 7: Concurrency

Reasons to use Distributed Systems:

Comparted to sequential systems, a Distributed System may provide

I an improved performance,I an improved failure tolerance,I or both.

Furthermore, implementing a system with heterogenous tasks as aDistributed System may be a more natural model than a strictlysequential implementation.Consider, e.g., an automotive system to control the engine and to check the road forobstacles and to display the relevant information to the user. Instead of a singleprocess handling all these tasks, this system is more naturally implemented by threedifferent processes with some amount of communication between them.


Concurrency is difficult. . . and often ends in the roadside ditch!

. . . or wherever


Questions

I Reliability / Availability:How can the system deal with individual nodes failing, or with thecommunication beteween certain nodes failing?

I Concurrency and Synchronisation:I Can several nodes work in parallel on the same task?I When do they need to synchronise?I How can they synchronise?

I Secure Communication:How to protect the system if one is using insecure communicationchannels (e.g., the internet, wireless communication, . . . )?(No topic for this lecture!)


Sequential↔ Concurrent

Two actions are sequential, if one can only be performed after theother. Examples:

I Make Coffee, drink it. (Sequential: drinking is only possible if thecoffee has been made before.)

I Drink Coffee and tea using the same cup. (Sequential, butundefined which comes first.)

I Drink Coffee and tea using two different cups. (Parallel: Alice maybe drinking her coffee while Bob is nipping from his tea.)


Alas: The Human Factor

I Humans are bad at thinking in parallel lines of action.I Intuitively, people assume that concurrent actions are performed

in the “correct” order.I This causes “Race Conditions”.I Most programming languages (including Ada) have a sequential

semantic:I If statement B follows statement A, then A has to be performed first

and B has to be performed then.I The optimizer may change this order, but the optimized program

must functionally behave as if A has been performed before B.I Programming languages with a parallel semantic have not been

successful, so far. (Some exception: purely functional languages.)I However, programming language with a sequential semantic can

still provide mechanisms for concurrent programs . . .


Support for Concurrent Progamming

I Some modern programming languages provide such support(Java’s “Threads”, Ada’s “Tasks”, . . . ).

I Otherwise, there are libraries providing such support (e.g.,pthreads). The main concept is the MutEx “Mutual Exclusion”.


A “Race Condition” ExampleA setuid-program

i f ( access ( f i l e , R OK) != 0) {e x i t ( 1 ) ;

}

fd = open ( f i l e , O RDONLY ) ;/ / read content o f fd . . .

The if-Statement sorts out users without read permission for the file.Readers with read permission can actually read the file.Where is the problem? A Time-of-check-to-time-of-use Attack:

1. Generate a file x which you are allowed to read!2. Start the program3. Change x into a symlink, which links to a file y for which you have

no read permission . . . but which you would like to read.With good timing (and, maybe, luck) you will actually read y . . .


Therac-25A computer-controlled radiation therapy machine

I Two modes for radiation therapy:direct mode radiation = dosage at the patientX-ray mode (for inner body parts; beam

passes through awolfram-“target”)radiation=dosage∗100

I predecessors have not been computer-controlled, but usedmechanical locks

I In use from 1983 on. Until 1987 several patients sufferd fromradiation poisoning, three killed.


Simulated Therac-25 Sessionfirst patient: 345 units in x-ray mode

−−−>d input dosage:345−−−>x (x−ray)−−−>r (radiate)

The machine needs 8 seconds to “warm up”.

The machine is done with “warming up”.

Radiation at patient # 1: 345 units


Simulated Therac-25 Sessionsecond patient: 321 units in direct mode

−−−>d input dosage:321−−−>y (direct)−−−>r (radiate)





Simulated Therac-25 Sessionthird patient: 312 units in direct mode

−−−>d input dosage:312−−−>x (x−ray)

Oh, the patient shall get direct mode and I selected X-ray.I’ll correct this!

−−−>y (direct)−−−>r (radiate)





Simulated Therac-25 Sessionfourth patient: 299 units in direct mode−−−>d input dosage:299−−−> x(x−ray)−−−>r (radiate)


Oh, the patient shall get direct mode and I selected X-ray.I’ll quickly correct this, while the machine is still “warming up”.

−−−>y (direct)



WTF?


Therac-25 Recall

direct mode radiation = dosage at the patientX-ray mode radiation=dosage∗100Implementation as two independent processes:

R to control the radiation, andM to control the mode and move the target.


Therac-25 Race Conditionfound in 1987 by accident

1. Operator entered correct dosage, but wrong mode (“X-Ray”).2. M selected X-ray; R calculated the radiation (= 100∗dosage)3. Operator corrected the wrong mode within less than 8 seconds.4. M switched into direct mode;

R started beaming – without re-calculating the radiation!5. Patient received 100 times more radiation than intended

(not always – a safety mechanism could trigger an early abort).

Short-term solution: Disable the key to correct wrong data!


7.1: Mutual ExclusionExtract from the pthreads-Tutorial:

Mutex

I . . . abbreviation for “mutual exclusion”. . . one of the primary means of implementing threadsynchronisation and for protecting shared data when multiplewrites occur.

I A mutex variable acts like a “lock” protecting access to a shareddata resource.. . . only one thread can lock (or own) a mutex variable at anygiven time.. . . if several threads try to lock a mutex only one thread will besuccessful. No other thread can own that mutex until the owningthread unlocks that mutex.

–367– S. Lucks Security Engineering 7: Concurrency 7.1: Mutual Exclusion

Mutual Exclusion (2)

I Mutexes can be used to prevent “race” conditions. An example ofa race condition involving a bank transaction . . .

Action 1 Action 2 Balancewithdraw 200.00 deposit 300.00 1000.00X := Read Account 1000.00X := X - 200.00 Y := Read Account 1000.00

Y := Y + 300.00 1000.00Write Account(X) 800.00

Write Account(Y) 1300.00

I In the above example, a mutex should be used to lock the“Balance” while a thread is using this shared data resource.



I Very often the action performed by a thread owning a mutex is theupdating of global variables. . . . the final value is the same aswhat it would be if only one thread performed the update. Thevariables being updated belong to a “Critical Section”.

I A typical sequence in the use of a mutex is as follows:1. Create and initialize a mutex variable2. Several threads attempt to lock the mutex3. Only one succeeds and that thread owns the mutex4. The owner thread performs some set of actions5. The owner unlocks the mutex6. Another thread acquires the mutex and repeats the process7. Finally the mutex is destroyed



I When several threads compete for a mutex, the losers block atthat call – an unblocking call is available with “trylock” instead ofthe “lock” call.

I When protecting shared data, it is the programmer’s responsibilityto make sure every thread that needs to use a mutex does so.


Simplifying the example

Thread 1 Thread 2a = Data; b = Data;a++; b−−;Data = a; Data = b;

Data may have been changed by either +1, 0, or −1.


Using pthreads-Mutex-Variables

Thread 1 Thread 2pthread mutex lock (&mut);

pthread mutex lock (&mut);/∗ Thread 2 blocks ∗/

/∗ critical ∗/ a = Data;/∗ critical ∗/ a++;/∗ critical ∗/ Data = a;pthread mutex unlock (\&mut); /∗ Thread 2 can finally continue ∗/

/∗ critical ∗/ b = Data;/∗ critical ∗/ b−−;/∗ critical ∗/ Data = b;pthread mutex unlock (&mut);

Data is the same as before – as it should be!


Mutual Exclusion: SummaryI Important concepts you should understand: “critical regions”,

“blocked process”, “mutual exclusion”.I Libraries like pthreads enable the programming of Concurrent and

Distributed Systems – even if the underlying programminglanguage doesn’t provide any such support.

I Note that critical sections must be found “by hand”, mutex-lock-/-unlock-operations must be inserted by the programmer, . . .

I The shared variable may be a complex data structure instead of asingle number.

I Different critical regious may overlap. This is complicated (inwhich order shall I call mutex unlock) and may cause “deadlocks”.

I All in all, the usage of mutex-variables is the “assembler language”approach towards concurrency.Secure and reliable software should follow a higher-levelapproach!


7.2: Example: Control a central heating

I User can set a target temperature.I The water pump

I is switched on if it is too cool in any room andI is switched off it is sufficiently warm in all rooms.

I The burnerI is switched on if the water in the boiler is too cool andI is switched off if the water is sufficiently warm.

–374– S. Lucks Security Engineering 7: Concurrency 7.2: Example

Spec of Package House

package House is

−− swi tch the pump on and o f fprocedure Pump Switch On ;procedure Pump Switch Off ;

−− read the temperature room−wisetype Cels ius is range −100 . . 150;type Sensor ID is range 1 . . 10;De fau l t : Ce ls ius := 3 ; −− f r o s t p r o t e c t i o nfunction Read Temp (Room: Sensor ID ) return Cels ius ;

−− user can increase or decrease the t a r g e t temperaturetype Button is (None , Plus , Minus ) ;function User Input return Button ;

end House ;


Spec of Package Boiler

with House ;

package B o i l e r is

−− swi tch on and o f f the burnerprocedure Burner Switch On is nul l ;procedure Burner Swi tch Of f is nul l ;

subtype P o s i t i v e C e l s i u s is House . Cels iusrange 1 . . House . Cels ius ’ Last ;

−− d e f a u l t f l ow temperatureDefau l t : constant P o s i t i v e C e l s i u s := 40;

−− ac tua l f low temperaurefunction Burner Temp return House . Cels ius ;

end B o i l e r ;


Implementation (Declarations)

with B o i l e r ; with House ;

procedure Cont ro l is

use type House . Cels ius ;

procedure B o i l e r C o n t r o l( Goal : B o i l e r . P o s i t i v e C e l s i u s ;

Plus Minus : B o i l e r . P o s i t i v e C e l s i u s := 2) is . . .

procedure House Control ( Target Temp : House . Cels ius ) is . . .

Target Temp : House . Cels ius := House . De fau l t ;

beginloop −− the i n f i n i t e ∗ c o n t r o l ∗ ∗ loop∗

. . .end loop ;

end Cont ro l ;


Implementation (Control Loop)

loop −− the i n f i n i t e ∗ c o n t r o l ∗ ∗ loop∗

−− f i r s t deal w i th user i npu twhile House . User Input /= House . None loop

i f House . User Input = House . Plus thenTarget Temp := House . Cels ius ’ Max( Target Temp +1 , 30 ) ;

else −− House . Unser Input = House . MinusTarget Temp := House . Cels ius ’ Min ( Target Temp−1, 3 ) ;

end i f ;delay 0 . 5 ; −−wai t f o r h a l f a second

end while ;

−− then deal w i th the b o i l e rB o i l e r C o n t r o l ( B o i l e r . De fau l t ) ;

−− then deal w i th the pumpHouse Control ( Target Temp ) ;

end loop ;


Implementation Boiler Control

procedure B o i l e r C o n t r o l( Goal : B o i l e r . P o s i t i v e C e l s i u s ;

Plus Minus : B o i l e r . P o s i t i v e C e l s i u s := 2) isuse B o i l e r ;Current : House . Cels ius := Burner Temp ;

begini f Current < Goal − Plus Minus then

Burner Switch On ;e l s i f Current > Goal + Plus Minus then

Burner Swi tch Of f ;end i f ;

end B o i l e r C o n t r o l ;


Implementation House Control

procedure House Control ( Target Temp : House . Cels ius ) isuse House ;Current : Ce ls ius := Cels ius ’ Last ;

beginfor I in House . Sensor ID loop

Current := Cels ius ’ Min ( Current , Read Temp ( I ) ) ;end loop ; −− Current = lowest Temp i n any roomi f Target Temp < ( Current−1) then

Pump Switch On ;e l s i f Target Temp > ( Current +1) then

Pump Switch Off ;end i f ;

end House Control ;


Busy Waiting

From Wikipedia:

In software engineering, busy-waiting or spinning is a technique inwhich a process repeatedly checks to see if a condition is true,such as whether keyboard input is available, or if a lock isavailable. . . .Spinning can be a valid strategy in certain circumstances, . . .In general, however, spinning is considered an anti-pattern andshould be avoided, as processor time that could be used toexecute a different task is instead wasted on useless activity.

(my emphasis)


Busy Waiting (2)

Beyond wasting processor time, there are further issues with thisapproach:

I One process (namely, the user input) can block the others.I Another process is critical: The burner must be switched of, after

some time.I Assume the user can’t make up her mind, and keeps one of the

the buttons pressed. Or, more realistically, one of the buttons ismechanically stuck in “pressed” state.If, at this point of time, the burner just happens to be switched on,the burner will not switch off, until . . .BOOOOM!


An Improved Solution

I We will represent our logical processas as three tasks in Ada.I Our tasks don’t terminate.I But when they have done their job, they will sleep (without

consuming CPU time), until the job needs to be done again. Weuse Ada’s delay-statement for this purpose.

I Our solution has still some weaknesses – ideally, our tasks shouldhave different priorities (e.g., controlling the burner should have ahigher priority than reading the user input).

I Ada provides the tools to assign priorities to tasks. But for thesake of simplicity, we will skip that issue here.


procedure Improved Control

with B o i l e r ; with House ;

procedure Improved Contro l isTarget Temp : House . Cels ius := House . De fau l t ;pragma atomic ( Target Temp ) ;pragma v o l a t i l e ( Target Temp ) ;

−− spec i f y th ree dependent taskstask B o i l e r C o n t r o l ;task House Control ;task User Input ;

−− the task implementat ionstask body B o i l e r C o n t r o l is . . .task body House Control is . . .task body User Input is . . .

begin −− main program can only te rmina tenul l ; −− a f t e r t e rm ina t i on o f a l l dependent tasks

end Improved Contro l ;–384– S. Lucks Security Engineering 7: Concurrency 7.2: Example

task body User Input

task body User Input isbegin

loopcase House . User Input is

when House . None =>nul l ;

when House . Plus =>Target Temp := House . Cels ius ’ Max( Target Temp +1 , 30 ) ;

when House . Minus =>Target Temp := House . Cels ius ’ Min ( Target Temp−1, 3 ) ;

end case ;delay 0 . 5 ; −− check about tw ice a second

end loop ;end User Input ;


task body Boiler Control

task body B o i l e r C o n t r o l is

use B o i l e r ;

Current : House . Cels ius := Burner Temp ;

beginloop

i f Current < Defau l t − 2 thenBurner . Switch On ; −− heat up !delay 0 .05 ; −− c r i t i c a l !

e l s i f Current > Defau l t + 2 thenB o i l e r . Burner . Swi tch Of f ; −− l e t i t coo l down !delay 5 . 0 ; −− not c r i t i c a l

end i f ;end loop ;

end B o i l e r C o n t r o l ;


task body House Control

task body House Control isuse House ;Current : Ce ls ius := Cels ius ’ Last ;

beginloop

for I in Sensor ID loopCurrent := Cels ius ’ Min ( Current , Read Temp ( I ) ) ;

end loop ; −− Current i s lowest temp i n any roomi f Target Temp < ( Current−1) then

Pump Switch On ; −− heat the rooms upe l s i f Target Temp > ( Current +1) then

Pump Switch Off ; −− l e t the rooms cool downend i f ;delay 5 . 0 ; −− check every f i v e seconds

end loop ;end House Control ;


Communication / Synchronisation

I Our processes communicate via a single global variableTarget Temp.

I If more than one process would write that variable, we would haveto worry about synchronisation.

I In our case, synchronisation is not an issue, since namelyUser Input is the only process writing to Target Temp.

I The remaining synchronisation issues are solved byI “pragma atomic (Target Temperature);” andI “pragma volatile(Target Temperature);”.

(What would be the problem if these pragmas where missing?)I Nevertheless, it would be better to perform synchronisation by

using a protected type.


Addition to Spec of House

package House is. . .

protected type Sync Cels ius isfunction Value return Cels ius ;procedure Increase ;procedure Decrease ;

privateC: Cels ius := De fau l t ;

end Sync Cels ius ;

. . .end House ;


Addition to Body of House

protected body Sync Cels ius is

function Value return Cels ius isbegin

return C;end Value ;

procedure Increase isbegin

C := Cels ius ’ Min (C+1 , 30 ) ;end Increase ;

procedure Decrease isbegin

C := Cels ius ’ Max(C−1, 3 ) ;end Decrease ;

end Sync Cels ius ;

v–390– S. Lucks Security Engineering 7: Concurrency 7.2: Example

Other changes

In procedure Improved Control:

→ replace “Target Temp : Celsius := House.Default;’ ’ by“Target Temp: Scync Celsius”.

In task body User Input:

→ replace “Target Temp := ...; ’ ’ by “Target Temp.Increase” or“Target Temp.Decrease”, respectively.

In task body House Control:

→ replace “Target Temp” in numerical expressions by“Target Temp.Value”.


7.3: Tasks in AdaI A task in Ada is a limited object (no assignment and no

comparison defined).I We can specify task types:

task type T is. . . −− s p e c i f i c a t i o n o f opera t ions

end T ;

I Then we have to provide the implementation:task body T is

. . . −− implementat ion o f opera t ionsend T ;

I And finally, we can generate objects of such a type:Al ice , Berta : T ; −− generates 2 task ob jec tsWorkers : array (1 . . 5) of T ; −− another 5 task ob jec tstype T Acc is access T ;Acc Worker : T Acc ; −− does not geneate any task ob jec t !

beginAcc Worker := new T ; −− yet another task ob jec t ( number 8)

–392– S. Lucks Security Engineering 7: Concurrency 7.3: Tasks in Ada

Properties of Ada-Tasks

I Once a task object has been generated, it runs. (It does not needsome start command.)

I The calling task can only terminate after the termination of alltasks depending on it.

I There is always an “environment task”. All other tasks dependeither directly or indirectly on the environment task. Theenvironment task (or “main program”, as one might also call it) canbe quite trivial, as in

beginnul l ;

end Improved Contro l ;


Ada-Tasks and Exceptions

I Within a task, raising and handling of exceptions works as usual.I But pay heed to the following: If an exception is not handled

within a task, the task will just terminate without any further action.I For beginners with programming Ada tasks – as most of you are –

this leads to a sometimes very surprising behaviour.I On the other hand, the obvious alternative to “no further action”

would be to propagate the exception to the calling task. But thatmight be the worse of two evils. The calling task would have to beprepared to handle any exception from any of the tasks it hadcalled at any time – driving the progammer crazy!

I Since Ada 2005, the calling task can define callback procedures,which will be called when a dependent task terminates.


Periodic Activities (1)

task Per iod i c ;

task body Per iod i c isbegin

loopAct ion ( ” Pe r iod i c ” ) ;delay 1 . 0 ;

end loop ;end Per iod i c ;

I Repeats Action every t + 1 seconds (at most), where t is the timeto perform the Action.

I No Busy-Wait.I The delay argument is from a predefined numerical type Duration.


Periodic Activities (2)

task More Per iod ic ;

task body More Per iod ic isI n t e r v a l : constant Durat ion := 1 . 0 ;use type Ada . Calendar . Time ;Next Time : Ada . Calendar . Time ;

beginNext Time := Ada . Calendar . Clock + I n t e r v a l ;loop

Act ion ( ” More Per iod i c ” ) ;delay u n t i l Next Time ;Next Time := Next Time + I n t e r v a l ;

end loop ;end More Per iod ic ;


7.4: DeadlocksDeadlocks can occur, when the following three conditions hold:

1. Some ressources are limited, i.e., if a process has to reserve theressource before using it, and other processes who need toaccess the ressource block until the ressource is given free.(The alternative is to have have so many ressources – or so fewprocesses – that processes never compete for access to aressource.)

2. Processes reserve ressources sequentially, in any order.(The alternative would be to require that a process reserves allressources it might ever need at its initialisation.)

3. There is no “preemption”, i.e., once a ressource has beenreserved by a process, it will not be given free except by theprocess itself.(The alternative would be to take away the ressource from theprocess, and the process would have to handle the sudden loss ofthe previously granted access to aressource.)

–397– S. Lucks Security Engineering 7: Concurrency 7.4: Deadlocks

Directed Graphs

I View a distributed system as a directed graph.I Vertices are processes and ressources.I If a process P has reserved a ressource R: edge P → R.I If a process P ′ waits for ressouce R to become available: R → P ′.I Any cycle of the form

P → R → P ′ → R′ → · · · → P

is a deadlock.I None of the processes P, P ′, . . . involved in a deadlock will ever

run again.I None of the ressources R, R′, . . . involved will ever become free

again.


Avoid Deadlocksor try to survive their occurance

I Finding potential deadlocks by testing is essentially hopeless.I Ideally, one tries to statically prove that a given Distributed System

cannot deadlock.I An interesting special case: Non-Blocking Synchronisation

(→Wikipedia: Non-blocking synchronization).I Often, one lifes with the existence of potential deadlocks, and their

occasional occurance. One still tries to1. discover the occurrence of a deadlock and2. handle the case by aborting (at least) one of the involved

processes, thus freeing the locked ressource(s).I Deadlock avoidance, recognition and handling is a research topic

in in computer science, with plenty of unsolved problems.


7.5: Communication between Tasks

I Data-Based communication:I “Mutual Exclusion” (see above)I Shared variables, such ase Target Temp aboveI Famous textbook examples: A semaphor, a monitor, . . .I Ada’s high-level concept: protected types

(recall the house control example, and see below for details)

I Communication by Message Passing:I Process S sends a message to another process R:

send Message to S – – no Ada syntax.I On the other hand, R waits for a message from S, or from any

processwait for Message from E – – not possible in Ada!wait for Message – – from anybody – – no Ada Syntax!

–400– S. Lucks Security Engineering 7: Concurrency 7.5: Communication

Guards

In general, the R will not wait for a single message, but for one ofseveral messages. Also, there are situations, where R cannot handlesome messages:

select – – Rendezvous – a bit like Adanot(Buffer Full) => wait for Write Into Buffer(Item);

ornot(Buffer Full) => wait for Read From Buffer(Item);

end select;

The abstract concept of a guard:

I If n ≥ 1 guards are true then any of the n alternatives is chosen (itis not specified, which one).

I It would be an error if, at any point of time, no guard is true!


Ada RendezvousI Ada-rendezvous are synchonous:

I Regardless of sender S or receiver R – whoever is first at therendez-vous point, has to wait for the other one.

I One could also imagine asynchronous communication:I S sends a message to R and then goes on with its own work

– without waiting for R to read the message.I Theoretically, synchronous and asynchronous communication are

equally powerful – one can be realized by the other one.Thus, most programming languages with support for concurrencysupport either synchronous or asynchronous communication, butnot both.

I Initially, Ada (83) supported only synchronous communication(message passing via rendezvous). This lead to too manywork-around solutions for problens which actually requiredasynchronous communication.Ada 95 introduced protected types for asynchronouscommunication. These are as powerfull as rendezvous.


Ada Protected Types – an Example

protected type Cnt is

entry Decr ;entry I n c r ;function Val

return I n t ege r ;

privateN: Natura l := 0 ;

end Signa l Ob jec t ;

If C is of type Cnt thenI tasks can call C.Incr;

C.Decr; or M:=C.Val;I without having to worry

about mutual exclusion.

protected body Cnt is

entry Decr when Val > 0 isbegin

N := N − 1;end Decr ;

entry I n c r when Val < . . . is. . .

function Valreturn I n t ege r is

beginreturn N;

end Val ;

end Cnt ;–403– S. Lucks Security Engineering 7: Concurrency 7.5: Communication

Some Hintsto consider before designing a concurrent system

Before designing a concurrent system, consider why you are designingthe system as a concurrent one, rather than as a sequential system:

1. for improved performance,2. for improved failure tolerance,3. because the real-world scenario the system is supposed to model

is inherently non-sequential.

There could be more than one of the above three reasons. But if noneof the three reason applies to you, you should consider to a sequentialdesign for your system.


Some Final Hintsfor designing a concurrent system

Are you following an approach, which avoids any deadlocks?If not, prepare how to handle deadlocks, when they occur!

Consider race conditions, time-of-check to time-to-use attacks, etc.Tryp to ensure that they don’t occur in your system.

Avoid busy waiting!

Try to use the high-level concepts for synchronisation andcommunication between processes, which are available to you(depending on the given programming languages and libraries you areusing).If possible, avoid low-level concepts, such as

I mutextes orI unprotected shared variables (even worse than mutexes)!


Documents

7: Distributed Systems and Concurrency fileA “node” can be a computer of its own, a core on a given processor, or even one of several processes on a single-core machine. In any