TheROME Programmers’Guide

May 1, 2001

LeslieJ.FrenchDistributedSystemsSoftwareGroup

CCRL, NECUSA4 IndependenceWay

Princeton,NJ 08540-6634

mailto:rome-admin@lists.sourceforge.net

You cangetthecurrent version of this document athttp://rome.sourceforge.net

ROME andthe ROME utilities arefree software; you canredistribute themand/ormodify themunderthe termsof the GNUGeneralPublic Licenseaspublishedby the FreeSoftwareFoundation;eitherversion2 of the license,or (at your option) anylaterversion.

They aredistributedin thehope thatit will beuseful,but WITHOUT ANY WARRANTY; without eventheimplied warrantyofMERCHANTABILIT Y or FITNESSFORA PARTICULAR PURPOSE. SeetheGNU GeneralPublicLicensefor moredetails.

You shouldhave receiveda copy of theGNU GeneralPublicLicensealongwith this program;if not,write to theFreeSoftwareFoundation,Inc.,59 TemplePlace- Suite330,Boston,MA 02111-1307

1

CONTENTS CONTENTS

Contents

1 Intr oduction 7

1.1 Summaryof Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.2 Theminimal ROME system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.3 Layout Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.4 Setting up theROME Environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.5 Project directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2 Tutorial I: Hello World 8

2.1 Step1. Build Input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.2 Step2. Generating theSourceTree. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.3 TheProgramSource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.4 Step3. Generating thetarget file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.5 Step4. Installing theImage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.6 Step5. Loading andExecuting theImage . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3 Tutorial II: Echo 12

3.1 Themoduledirectory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.2 Thesourcefile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.3 Setting up theProject . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.4 Creating theModule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.5 Build . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.6 Make,Load,Run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4 Tutorial III: Perf 16

4.1 PerfFunctionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

4.2 ThePerfMessageset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

4.3 Generating theSourceTree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.4 TheSourceCode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.4.1 Theuptime process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.4.2 Theuptime timeoutroutine. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4.4.3 Datafor theuptimeprocess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4.4.4 ThePerfmainprocess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4.4.5 TherespQueueHandler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

2

CONTENTS CONTENTS

4.4.6 Therespmainprocess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.4.7 Therespprocedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.4.8 Theperf timeoutroutine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.4.9 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.4.10 TheRTB entry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.5 Running theimage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

5 Tutorial IV: timer_pc 25

5.1 Timer Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

5.2 TheTargetFile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

5.3 TheRTB File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

5.4 Thetimerlib sharedlibrary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

5.4.1 Local Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

5.4.2 Thetimeout routine. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

5.4.3 Theuntimeoutroutine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

5.4.4 Sleep . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5.4.5 Thetimer_tmhandler routine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5.5 TheClock routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

5.5.1 Thetimer init routine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

5.5.2 TheQueueHandler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

5.5.3 TheMain Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

5.5.4 Theinterrupt handler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

5.5.5 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

5.6 Modifying theClock Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

5.6.1 Making thesource file writable. . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

5.6.2 Changing thesource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

5.6.3 Running theimage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

6 Tutorial V: UART16550 34

6.1 mblks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

6.2 Themessageinterface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

6.2.1 Control Messages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

6.2.2 Moving Datawith Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

6.3 Applicationto theUART16550 driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

3

CONTENTS CONTENTS

6.4 TheTargetFile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

6.5 SourceCode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

6.5.1 Theinit routine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

6.5.2 TheInterrupt Handler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

6.5.3 TheOPENMessage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

6.5.4 TheCLOSEMessage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

6.5.5 TheFLUSHMessage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

6.5.6 TheFETMBLK Message. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

6.5.7 TheGETMBLK Message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

6.5.8 TheNEWMBLK Message. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

6.5.9 TheOUTMBLK Message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

6.5.10 ThePUTMBLK Message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

6.5.11 TheRETMBLK Message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

6.5.12 Themainroutine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

6.6 TheDebugging Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

6.6.1 Philosophy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

6.6.2 Enabling a debugenvironment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

6.6.3 Debug commands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

7 Tutorial VI: Ethernet ARP 51

7.1 Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

7.2 FILEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

7.3 URLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

7.4 Upstream,Downstream andQueues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

7.5 ConfigurationMessages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

7.6 Finite StateMachines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

7.7 Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

7.8 SourceCode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

7.8.1 DataStructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

7.8.2 TheCommandMessage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

7.8.3 TheEVENT Message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

7.8.4 TheEVENT Reply . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

7.8.5 TheOPENMessage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

7.8.6 TheCLOSEMessage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

4

CONTENTS CONTENTS

7.8.7 TheFLUSHMessage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

7.8.8 TheFLUSHReply . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

7.8.9 TheFETMBLK andGETMBLK Messages . . . . . . . . . . . . . . . . . . . . . 60

7.8.10 TheFETMBLK andGETMBLK Replies . . . . . . . . . . . . . . . . . . . . . . 61

7.8.11 TheNEWMBLK Message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

7.8.12 TheNEWMBLK Reply . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

7.8.13 TheOUTMBLK andPUTMBLK Messages . . . . . . . . . . . . . . . . . . . . . 63

7.8.14 TheOUTMBLK andPUTMBLK Replies . . . . . . . . . . . . . . . . . . . . . . 64

7.8.15 TheRETMBLK Message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

7.8.16 TheRETMBLK Reply . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

7.8.17 TheTIMEOUT Reply . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

7.8.18 Themainprocess. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

8 Tutorial VII: Network Echo 66

8.1 Choosing theModules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

8.2 Initialising IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

8.3 Source Code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

8.4 Build, LoadandRun . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

9 Legacy Applications and the C Runtime 68

9.1 TraditionalBuffering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

9.2 Buffered Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

9.2.1 printf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

9.2.2 fputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

5

LIST OFFIGURES LIST OFFIGURES

List of Figures

1 Module Structure for Tutorial I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2 Module Structure for Tutorial II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3 ModulesStructure for Tutorial III . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4 Module Structure for Tutorial VII . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

6

1 INTRODUCTION

1 Intr oduction

This documentpresentsa structuredwalk-throughof the useof the ROME Operating System. It givesexamples of useof theexternal support tools including theROME TargetBuilder (RTB). It describestheAPI to the kernel andto the dataflow abstractions implemented in the Run-Time library. The documentalso usesa numberof ‘hands-on’ examplesof modules in a tutorial manner, gradually to integrate thevariousstandardcomponentsof ROME.

1.1 Summary of Features

A complete ROME system is built up from a numberof modulesaccording to the required application.Somemodules arespecific to a particular piece of hardware(the cpu plug-in, anddevice drivers), somearemachine independent andusableacrossawiderangeof applications(for exampletheROME coreandtheConsolemultiplexer) andsomeareapplication specific.

Internally, ROME usesa message-passing systemfor inter-process communication. ROME definesastandardsetof messages,but modulesareabledefinetheir own messagesfor specific functions.Most ofthe details of individual messages arehidden from high-level programsby interfacelibraries (includinganimplementation of thestandard‘C’ runtimes).However lower-level modules(including devicedrivers)mustunderstandtheformatof messages in order to processthem.

TheROME systemis linkedinto asinglenaming(andaddressing) space.Thismeansthatmostinterfacesareavailable through sharedlibraries,but placesrestrictionson thevariable nameswhich maybeused.

To managethis system the ROME toolkit containsa configuration util ity rtb. Compilation is performedusing gccasa cross-compiler, with theappropriate assemblerandlinker.

Whendescribedin ‘Reference Manual’ format this combinationof tools, environment, librariesandin-terfacesmay appear daunting, though the underlying principles aredeliberately simple. This documentseeks to overcomethatobstacleby presentinga walk-though of ROME featuresin a sequenceof workedexamples, which canbebuilt, loaded,andexecutedon readily-availablehardware. This guideshould beusedin conjuction with the otherROME manuals andthe referencepages for all the modulescurrentlyavailablein ROME.

1.2 The minimal ROME system

Thesmallest ROME systemthat it is possible to construct contains five components:

1. The‘system’ sharedlibrary, implementing theeightbasiccoreroutinesfor themessage-passingandmemorymanagementfunctions,andthe“idle” process,runby theschedulerwhennootherprocessis runnable;

2. The CPU-dependent plug-in, containing the context switching codefor inter-processcommunica-tion;

3. Theinterrupt dispatcher;

4. Thesystem-level debugger(optional,but highly recommended);

5. The‘C’ runtime library andcoreinterfacelibrary for theStandardMessageSet.

Thesefiveelements areshown aspartof theROME basesystemin theprocessdescription figures.

7

1.3 LayoutConventions 2 TUTORIAL I: HELLO WORLD

1.3 Layout Conventions

In the tutorial sections,examples of user input areused. Suchinput is distinguishedby a bold type-

writer font. Systemoutput is in regular weight. ‘C’ examplesareformatted in regular andbold weights.

The examples of system builds, anddirectory listings arecopies‘asis’ from testsystemsusedto verifythebuild. Naturally, theoutput producedat different timeson differentmachineswill causethis output tovary.

1.4 Setting up the ROME Envir onment

Beforeyou canrun theprogramsin thetutorial, you have to install theROME system on a local machine.TheROME environmentis dividedinto anoptionalCVSsourcetree, local ‘project’ directoriesandtoolkitbinaries. At this stageis it assumedthat you have already downloadedandinstalled the ROME TargetBuilder (RTB) according to the RTB UserManual, andhave its binary on your search path. Theprojectdirectoriescanbelocatedanywhere in a (writable) filing system

Youmayneedto consult the“GettingStarted”guide,or your localadminstratorto find outhow theROMEsourcesaremanaged in your environment. The sourceswill eitherbe in a local copy of the treeandbeaccessible through the RTB import commandor be in a CVS repository andbe accessiblethrough theRTB CVSdialogues.Beforeyou start, you will need to configure your personal RTB preferences.

All theseexamples aredesignedto run on an Intel-basedPC, with at leasta 486-classCPU. It is alsoassumed thatthedevelopmentplatform is PC-based.Thetutorials bootdirectly from a floppy disk on thetarget machine(which meansthat the BIOS on the target musthave ‘boot-from-floppy’ as the primarybootoption). All theoutput is directedto a9600-baud serial line at0x2f8 (usually ‘COM1’). Thesupporttoolkit contains a program,xtip, that canbe usedto display this output andsendinput keystrokes to thesystem. Thereasonstheexamplesusetheserialinterface(rather thanthePCkeyboardandscreen) aretosimplify theprograms,andbetter to represent‘real’ embedded applications,which areusually debuggedthrough a UART interface,rather thananattachedscreenandkeyboard.

1.5 Project dir ectories

ROME systemsarebuilt within a host filing system, in a personal or project directory. RTB usesthisdirectory as the place in which to create a new system, and is designed to work with a completelyempty initial directory. However, the configuration information for eachproject mustbe stored some-where. Thesetutorials assumethere is a single directory being usedfor all these projects. The top-level of this directory holdsproject informationandtheindividual projects arecreatedin sub-directories.At this point you should set the projects field in the RTB Preferences dialog to this directory (e.g./home/leslie/rome_tutorial). This completesthe environment setup, asRTB automatically createsall di-rectoriesthatareneeded.Onceyou have donethis,you canmove into theprojectsdirectory:

cd rome_tutorial

2 Tutorial I: Hello World

This first tutorial usespre-definedsources to introducetheconfiguration andcompilation mechanism. Itbuildsasimpleloadablesystemwhich printshello world on theconsole output. Theprocessstructurefor

8

2 TUTORIAL I: HELLO WORLD 2.1 Step1. Build Input

this systemis shown in figure1. Apart from theminimal coredescribedin chapter1, this system includesa driver for thesystem’s UART (serial line). Two non-driver processescomplete thesystem, theconsoleprocessandthehello process.

Core Interface API

core/"idle"cpu-plugin

icu-plugin

C Runtime Library

debugger

logical message path

actual message path

ROME base system

hello world

Console

UART IRQ

Figure1: ModuleStructurefor Tutorial I

2.1 Step1. Build Input

Thefirst stage is to createa project description using RTBin thedirectory created in chapter1.

In RTB, createa new project description for the hello world program andcheckout (or import) the fol-lowing modules, according to the CVS hierarchical structure (if you areimporting the modules,you donot have to provide thedottedclass ID prefix):

Target: SPB450

MessageSets:Standard, Console

Modules: Kernel.PlugIn.CPU_I386,Driver.Interrupt.ICU_I386,Kernel.ROME, Library.ROME_IF, Driver.Serial.SERIAL_UART16550,Application.System.CONSOLE, Library.CLIB, Application.Demo.HELLO

This describesthe basic hardware configuration andbringsin the corescheduler anddrivers needed forthetutorial. Whenexamining theindividual processes, notethat thehello Processline hasthe‘usestdio’flag set,in contrastto theotherprocesses.

9

2.2 Step2. Generating theSourceTree 2 TUTORIAL I: HELLO WORLD

2.2 Step2. Generating the SourceTree

Click on the ‘GenerateBuild File’ button to run the system builder program andthencheck the dialogpopup for any errors. Theresult should bea top-level directory listing looking something like this:

cd hello_worldls -ltotal 7drwxr-xr-x 4 leslie leslie 1024 Apr 2 17:30 MessageSetsdrwxr-xr-x 13 leslie leslie 1024 Apr 2 17:35 Modules-rw-rw-r-- 1 leslie leslie 3435 Apr 2 17:34 Project.rtbdrwxr-xr-x 3 leslie leslie 1024 Apr 2 17:30 Targets

Eachof the componentstypes hasa sub-directory below the project directory, which also contains theproject description file.

cd Modulesls -ltotal 14drwxrwxr-x 3 leslie leslie 1024 Apr 2 17:35 CPU_I386drwxrwxr-x 3 leslie leslie 1024 Apr 2 17:35 Clibdrwxrwxr-x 3 leslie leslie 1024 Apr 2 17:35 Consoledrwxrwxr-x 3 leslie leslie 1024 Apr 2 17:35 Hellodrwxrwxr-x 3 leslie leslie 1024 Apr 2 17:35 ICU_I386-rw-rw-r-- 1 leslie leslie 1849 Apr 2 17:35 Makefiledrwxrwxr-x 3 leslie leslie 1024 Apr 2 17:35 ROMEdrwxrwxr-x 3 leslie leslie 1024 Apr 2 17:35 SERIAL_UART16550drwxr-xr-x 2 leslie leslie 1024 Apr 2 17:35 Symbolsdrwxr-xr-x 2 leslie leslie 1024 Apr 2 17:35 includedrwxr-xr-x 2 leslie leslie 1024 Apr 2 17:35 init-rw-rw-r-- 1 leslie leslie 873 Apr 2 17:35 ld.inputdrwxrwxr-x 3 leslie leslie 1024 Apr 2 17:35 rome_if

In theModulessub-directory, eachsourcemodulehasadirectory createdfor it, andthereis anconfiguration-specific include directory for the header files appropriate to this build. The other directories, init andSymbols, arecreatedto hold theprocess-specific initialisation filesandthedebuggersymboltable,respec-tively.

Finally, Makefile andld.input areinput files to thefollowing stages.

2.3 The Program Source

Beforeproceedingto compilethesystem,thissectionshowsthesourcefile usedastheapplicationto printhello world. Thefile is hello.c in theHello sub-directory.

ls -l Hellototal 5drwxrwxr-x 2 leslie leslie 1024 Apr 2 17:34 CVS-r--r--r-- 1 leslie leslie 1001 Aug 26 2000 Hello.rtb-rw-rw-r-- 1 leslie leslie 851 Apr 2 17:35 Makefile-r--r--r-- 1 leslie leslie 1759 Jul 18 2000 hello.c

10

2 TUTORIAL I: HELLO WORLD 2.4 Step3. Generating thetarget file

As this modulewasretrivedfrom thelocal CVSrepository, thesourcefile is a read-only copy of themainCVSfile. It contains thefoll owing sourcecode:

#include <stdio.h>

void hello_process(void){

printf("hello world\n");}

This is essentially a standard‘C’ programusingtherun-time library printf routine. OnedifferencefrommostUNIX implementationsof thesameprogramis thatROMEdoesnotusetheargcandargvparametersto processes. Also, sinceROME uses a single shared namespace,routine nameslike main are rarelyappropriate. Most externally-v isible namesarecomposedof a modulename(hello, here) anda module-specific local name.Themainexceptionsarestandardroutines(like printf in theruntimes)andvariablesdeclaredto bestatic.

2.4 Step3. Generating the target file

Thedirectory is setup ready to run themake utili ty (which is not partof theROME toolkit, but should beon a utilit iespathfor theshell). Notethat theexact output mayvary betweenbuilds:

makecd CPU_I386; make. . .cd ROME; make. . .. . .cd Symbols; make. . .ld -T ld.input -Map=target.map

make runstheC compilerandassemblerin eachof themodulesubdirectories,thenruns ld to producethetop-level target. This addstwo files to themaindirectory:

-rwxrwxr-x 1 leslie leslie 87363 Apr 2 17:59 target-rw-rw-r-- 1 leslie leslie 19172 Apr 2 17:59 target.map

target is the downloadable image,and target.map is the linkage addressmapandsymbol table for thatimage.

2.5 Step4. Installing the Image

In order to downloadthe image,it mustbe copied onto a bootablefloppy disk, with a boot loader at thestartof thefile. This is done usingthe install rule in theMakefile:

11

2.6 Step5. Loading andExecuting theImage 3 TUTORIAL II: ECHO

make install. . .getprog target target.b

sh[1]: name .text type PROGBITS flags 6 addr 10000 . . .sh[2]: name .rodata type PROGBITS flags 2 addr 198a0 . . .

Pad 15 bytessh[3]: name .data type PROGBITS flags 3 addr 1da00 . . .

(dd if=/p/sa/rome/v1.0/Tools/bin-8086/boot1.b bs=512 conv=sync;dd if=target.b) > boot.b

0+1 records in1+0 records out138+1 records in138+1 records outdd if=boot.b of=/dev/fd0 bs=512 conv=sync139+1 records in140+0 records out

Thegetprog program(suppliedaspartof theROMEtoolkit) extractsthebinarypartsof theimagefrom thetarget file. Thefirst dd commandpads this out to anintegral number of sectors.Thesecond dd commandaddsthe boot loader to the front andwrites the whole imageout directly to a floppy disk. The boot.bloader is alsosupplied in thetoolkit.

2.6 Step5. Loading and Executing the Image

The floppy disk should be placed in the target machineand the machinereset (or poweredon). Thefollowing output should appear on thextip console.

ROME Initialising.Copyright 1997 NEC USA Inc.Built by leslie on pc-rome.ccrl.nj.nec.com at Mon Apr 2 18:21:14 2001Starting the Scheduler.....hello: hello world

hello: terminated

Thelines areprintedby thedownloadedimage. The‘Built’ line giveslists theusername,hostname,dateand time at which the build was invoked from RTB to generate the configuration for this image. Thisinformationhelpsensurethatthecorrect imageis beingloaded.

The ‘hello world’ string is precededby the nameof the processwhich generatedit. This is oneof thefunctionsof the‘Console’ moduleincludedin theimage.

3 Tutorial II: Echo

The previous tutorial usedonly pre-definedsource modulesto produce an image. In this tutorial a newapplication-level moduleis created,showing how themoduleis integratedinto thebuild environment. Thismodule, Echo, will just copy linesof input on stdin to stdout. However, themainpurposeof this moduleis to describe integratinga new component into the ROME build environment. The process structureisshown in figure2. The only differencebetweenthis figure and the previous tutorial is that the “hello”processis replacedby the“echo” process.

12

3 TUTORIAL II: ECHO 3.1 Themodule directory

Core Interface API

core/"idle"cpu-plugin

icu-plugin

C Runtime Library

debugger

logical message path

actual message path

ROME base system

Console

UART IRQ

echo

Figure2: ModuleStructurefor Tutorial II

3.1 The module dir ectory

The namesfollowing the Get or Import commands in the RTB dialogues arethe namesof modules, forexampleCPU_I386andHello. In ROME, a module is just a directory within thefiling system.Modulesmustbecopiedinto theprojectdirectory before they canbeusedin theproject.

For this tutorial, create a new directory under the rome_tutorial directory called Sourcesand createasub-directory within it calledEcho. This will bethedevelopmentdirectory for thenew module.

mkdir rome_tutorial/Sourcesmkdir rome_tutorial/Sources/Echocd rome_tutorial/Sources/Echo

This structure, with individual moduledirectories under a top-level directory, foll ows the pattern of theROME CVStreearrangement.

3.2 The sourcefile

The executable codeof this example is quite straightforward. Using the standardC runtime routinesabuffer is alternately filled andemptied. Theapplicationrunsasa single ROME processfrom theroutineecho_process. Thecompletesourcefor theecho.cfile is thenasfollows:

#include <stdio.h>

void echo_process(void){

13

3.3 Settingup theProject 3 TUTORIAL II: ECHO

char buffer[256];while (fgets(buffer, 255, stdin)){

fputs(buffer, stdout);}

}

Createthis file usingyour favourite edit, andthis completestheEchomodulesource(easy, isn’t it).

3.3 Setting up the Project

This stepis almostthe sameas in the previous tutorial. Start rtb from the Sources/Echo directory andcreate a new project Echo. Checkout (or import) the samesetof of componentsasin tutorial I, exceptfor the Hello module (if you areimporting the modules, you do not have to provide the dotted classIDprefix):

Target: SPB450

MessageSets:Standard, Console

Modules: Kernel.PlugIn.CPU_I386,Driver.Interrupt.ICU_I386,Kernel.ROME,Library.ROME_IF, Driver.Serial.SERIAL_UART16550,Application.System.CONSOLE, Library.CLIB

3.4 Creating the Module

Stayingwithin the Echo project in RTB, click on the New Module option, then edit the NewModule0modulein theModuleslist asfollows:

1. Change themodule nameto Echo

2. SettheClassID to Application.Demo

3. Under‘SourceFiles’ addaC file andselect echo.c (assuming you startedRTB in theSources/Echodirectory). Selectthe ‘Copy to module’ option. You could also createthe Echomodule and theecho.c file directly in Module/Echo in which caseyou would usethe‘Just addfile’ option.

4. Under‘Processes’adda new processnamed‘echo’. Remove the entriesfor the Init Function andQueueHandler, leavingonly theMain Function named‘echo_process’. Setthepriority to 1 andthestack sizeto 4k and(mostimportant) check the‘Processusesstdio’ box.

This completesthecreation of thenew module for theproject. Thestatus of themoduleshould be‘localonly’ in theproject listing. Checktheboxto theright of theCVSstatusto includethemodulein thebuild.At this point, theModules/Echo directory in therome_tutorial/Echo treecontainstwo files:

total 2-rw-rw-r-- 1 leslie leslie 939 Apr 3 12:19 Echo.rtb-rw-rw-r-- 1 leslie leslie 155 Apr 3 12:15 echo.c

Thesource file wascopied from thedevelopmentdirectory, andtheEcho.rtb file wascreatedby RTBtohold theproject description. In this case,thesourcefile is writable, becauseit wasnot createdthrough aCVScheckout.

14

3 TUTORIAL II: ECHO 3.5 Build

3.5 Build

As before,click on theGenerateBuild File button in RTB, producing thelocal moduledirectoriesandthetop-level commandfiles. Mostly thesearethe sameasthe previousexample,but thereis now an Echodirectory andno Hello directory.

We can tracethrough someof the build procedure for this module using the files in thesedirectories.Towardstheendof theOutput Summarypopupis therecordof theprocessingdonefor theEcho module:

Creating MAKEFILE ’/home/leslie/rome_tutorial/echo/Modules/Echo/Makefile’

Thebuild operationhasalsogeneratedanentry in thetop-level Makefile for theEchomodule:

mod_Echo: Echo/echo.ccd Echo; make

Thedependency list is derivedfrom thelist of files in theEcho.rtbfile.

Within the Echo directory, the Makefile containsa dependency on the echo.o file, derived from the ‘C’sourcefile nameadded to themoduledescription. Therestof theMakefile is a setof rulesusedto controlthecompilation. Thesedepend on thetarget CPUfor theproject, andcomefrom theTarget/SPB450file.Settingup theserulesis describedin theROME Porting Guide

The procedure is completedby the directive in the ld.input file to include the object in the final targetimage:

INPUT ( Echo/echo.o )

All that remainsto complete the chain is to start the processwithin the executable image. The Processinformationin the RTB dialoguehasbecome a initializer in a datastructurewhich is compiled into theimage.This is the init.c file in the init module,which is createdautomatically during build:

extern void echo_process(void);. . .

{"echo", 0, echo_process, 0, 1, 4096, 1, 0},

Although thereareother options anddirectiveswhich maybeused,all modulesfoll ow this samepatternfrom a sourcetreeinto a local directory with entries in theMakefilesandtheprocessinitialisation table.

3.6 Make,Load, Run

The target is madeusingmake asbefore, andthencopied onto a floppy usingmake install , andbootedinto thetestmachine:

ROME Initialising.Copyright 1997 NEC USA Inc.Built by leslie on pc-rome.ccrl.nj.nec.com at Tue Apr 3 12:45:27 2001Starting the Scheduler.....

15

4 TUTORIAL III: PERF

At this point the system is waiting for input. All keyboardinput goesthrough the “console” process. Inthepreviousexample,consoleoutput wasshown, with theprocessnamebeingpre-pendedto output lines.Thesameschemeworksfor input, thefirst token on theline is usedasa processname.

echo fooecho: fooecho this is a test of the echo systemecho: this is a test of the echo system

andasbefore,output appearswith theprocessnamebeforetheline.

Theprocessis in a continuousloop waiting for input, it never terminatesandsothe‘terminated’messageseenin theprevioustutorial does not appear.

4 Tutorial III: Perf

This tutorial introducesthe ROME messaging system using the low-level performancemeasuring tool,Perf. Although Perf is an important tool for categorising system performance(seethe useof Perf inthe ROME Porting Guide), the main focus of the description here is on how it handles inter-processcommunication.

The real key to the ROME system lies in efficient inter-processcommunication. The system scheduleris built around a request–response messaging system.A processwishing to usethe servicesof anotherprocessin thesystemlocatesthedestination processusingthecoreroutine rome_find_queue. Thesendingprocessgeneratesa request in the form of a standard messageblock and queuesit for the destinationprocessusingthecoreroutinerome_send_message. Thedestinationprocessreceivesthemessage into itsprocessingloopby calling rome_await_message. Onceit hashandledtherequest,themessageis returnedto theoriginatorusingrome_reply, a coremacrowhich calls rome_send_message to enqueuethereply totheoriginator. Theoriginator receivesthereply with rome_await_message andthecycle is complete.

In somecases, requestsmay require very little immediateprocessingby the receiver. Examplesincludeadding a ‘read’ requestto aqueueof requestsawaiting thearrival of data, or ‘status’ messages thatcanbeanswered immediately. For these operations theoverheadof thepair of context switchesfar exceeds theprocessingneededto handle therequest. ROME providesa ‘f astpath’ interfacefor a receiving processtohandle such messagesin the context of the sending process.This routine, termeda ‘QueueHandler’, iscalled asa subroutine from the corebefore the message is added to the destination process’ queue. Thehandler mustreport oneof threeactionsbackto thecore: (a) themessagehasbeencompletelyprocessedby thehandlerandnofurther actionis required; (b) themessagehasbeenprocessedby thehandlerandthedestinationfield updated,thecoreshould passthemessageto thenew process;or (c) themessage shouldbequeued for themainprocess.

One frequent operation for a queue handler to perform is to add a message to a queuedatastructureshared with the main process(or an interrupt handler). In these cases,the queue handler often requiresan interlock on the queue, to prevent simultaneous updateswhich may destroy its integrity. TheROMEcore implements a single form of interlock, the critical section, which is a block of codeguaranteedtorun to completion without pre-emption by another process or handler. Sincethe rome_send_messageroutine, which calls the handler, runs as a critical section to protect the systemscheduler queues, theQueueHandlerroutine automatically runs interlocked.

16

4 TUTORIAL III: PERF 4.1 PerfFunctionality

As an alternative to messaging, for operations that can complete immediately, a simple shared-libraryinterfacemay suffice. This incurs only the overhead of a procedurecall andreturn. In caseswherethelibrary mustaccessshareddata,the routinemustrun in a critical section. This addstheoverheadof onecall eachto rome_start_critical androme_end_critical.

Depending on thearchitecture of themachinea particular function maybebetter implementedwith onesolution from this range rather thananother. For example,the QueueHandleroverhead is addedto allmessagessentto theprocess.If thehandleronly processesvery few messagesoutof thetotalnumber sentto theprocess,it maybelessexpensive,in thelong term,to omit thehandler functionandallow thosefewmessagesto passto themainroutine.

ThePerf processis usedto calibratea systemfor these costs.It is usedhereto introducethemessagingconceptsdescribedabove andasanimportant introduction to thetiming andperformanceissuesthatwillgovernsomeof thedesign of a full system.

4.1 Perf Functionality

Perf measurestheamountof realtimetaken for variouslow-level operationswithin thecoreof theROMEsystem. It usesthesystem timermoduleto requestamessageevery10seconds. In each10-secondintervalit performsoneof asetof four testsby counting thenumberof timesamessagecanbepassed throughthesystem beforethe10secondselapses.A simpledivisionthen giventheelapsedtimefor asingleoperation.Thefour operations are:

(a) Sendinga message which is returned immediately by the QueueHandler, andreceiving thereply.

(b) Sendinga message to the queue of the destination process,which generatesthe reply, andreceiving thereply (2 context switches)

(c) Operation (b) plusa single routinecall

(d) Operation (b) plusa routinecall which runsa critical section.

Subtraction of (c) from (b) givestheroutine call overhead,subtraction of (d) from (b) givestheoverheadfor a protectedroutine call. Obviously, measurementsobtainedfrom thesubtraction of two similar num-bersarenot particularly accurate,however, theresults arequite consistent over a wide range of machinesandarchitectures.

Thereis alsoa longer-running timer processincluded in thePerf module. By measuring thereal time ofthesystem, theinterval timercanbecalibratedoveranumberof hours(for exampleovernight). Thisgivesa measure of theaccuracy of theintervaltimer on which thesemeasurementsarebased.

Theprocessstructureis shown in figure3.

4.2 The Perf Messageset

Individual messagesareidentified by in ROME two numbers, onefor themessage andonefor thereply.Message numbersaregeneratedby the RTB tool whenmessages aredefined. Thereis a ‘standard’ setof messages pre-definedin ROME for managing dataflows; theseare the subject of the tutorial V. In

17

4.2 ThePerfMessageset 4 TUTORIAL III: PERF

Core Interface API

core/"idle"cpu-plugin

icu-plugin

C Runtime Library

debugger

UART IRQ

resp perf uptime

logical message path

actual message path

ROME base system

Console

timer

Figure3: Modules Structure for Tutorial III

this tutorial a pair of privatemessagesareusedbetweentwo processes, anda message definedby anotherprocessis used throughalibraryinterface.Becausemessagesareusedto communicatebetweenprocesses,many modules maysharethesamemessage definitions. Sincemessages do not ‘belong’ to processesinany obviousway, thedefinitionsof messagesaregroupedinto separatecomponentscalledMessage Sets.Here,two privatemessagesaredefinedin theMessage Setfor Perf. ThemessagesarecalledTESTQandTESTCX, andeachtakes a single (dummy)argument. This moduleis available in themastersource tree(Modules/Perf), andthemessagesetis in MessageSets/Perf.

Message TESTQ{

int arg1;}Message TESTCX{

int arg1;}

Thearg1 fieldsarenot used in this example.

Whenthis is processedby RTB, thefour symbols ROME_M_TESTQ, ROME_M_TESTCX, ROME_R_TESTQandROME_R_TESTCXaregeneratedto identify themessages andthecorresponding replies,andthepa-rameterlists areturnedinto type definitions.

18

4 TUTORIAL III: PERF 4.3 Generating theSourceTree

4.3 Generating the SourceTree

Thistutorial usesonly existingmodules.Createanew projectPerf, andcheck out(or import) thefollowingsetof of components(if you areimporting the modules,you do not have to provide the dotted classIDprefix):

Target: SPB450

MessageSets:Standard, Console,Perf,Timer

Modules: Kernel.PlugIn.CPU_I386,Driver.Interrupt.ICU_I386,Kernel.ROME, Library.ROME_IF, Driver.Serial.SERIAL_UART16550,Driver.Timer.timer_pc, Application.System.CONSOLE,Library.CLIB, Application.Demo.Perf.

Whenthe project is built, aspart of the build, the file Messages.his created in the include directory. Attheendof this file arethedefinitionsuseby Perffor its messaging operations:

typedef struct{

int arg1;} ROME_T_TESTQ;#define ROME_M_TESTQ 0x00020102#define ROME_R_TESTQ 0x00020103typedef struct{

int arg1;} ROME_T_TESTCX;#define ROME_M_TESTCX 0x00020104#define ROME_R_TESTCX 0x00020105

Replycodesaredistinguishedfrom thecorresponding requestby having theROME_REPLY_BIT (0x01)set.Notethatrequestshaveevencodes,andreplies have oddcodes.

4.4 The SourceCode

The codefor Perf will be presentedin smaller sections to emphasisethe functionality of the individualroutinesand procedurecalls. The system comprises three processes: “perf” actsas originator for thevariousmessages; “resp” is theresponderprocess;and“uptime” measuresthe length of time it hasbeenrunning. Of thethree,“uptime” is thesimplestandwill bedescribedfirst.

4.4.1 The uptime process

This processwakesup once every 60 seconds:

void up_process(void){

timeout(up_timer, (caddr_t)0, 60*HZ);

The documentation for the Timer moduledescribes the shared library interfacefor requesting intervaltimeouts. The timeout routine sendsa messageto thetimer module.Thereply is delayedfor theamountof time specified on thecall.

19

4.4 TheSourceCode 4 TUTORIAL III: PERF

while (1==1){

ROME_MESSAGE *mp = rome_await_message(0, 0);

So,having sentaninitial timeoutrequest, theprocessjust sleepsinsiderome_await_message waiting forthereply. Thereply comesasthereturn value from rome_await_message.

switch (mp->opcode){

case ROME_R_TIMEOUT :

The opcodefield in the message identifies the message type. The switch statementhandles the (only)message typereceivedby this process(ROME_R_TIMEOUT)

timer_tmhandler(mp);timeout(up_timer, (caddr_t)0, 60*HZ);break;

Usually the timeout mechanism is usedto call a registered routine, in this caseup_timer, when thetimer expires. This is implemented by the timer_tmhandler routine. In a production system, the callto timer_tmhandler is usually hidden behind a generic messagehandler mechanism (explainedin tutorialV), but hereit is madeexplicit. The main processcalls the timer_tmhandler routine asrequired,beforere-issuing thetimeout request.

}}

}

Finally, all this is enclosedin a ‘forever’ loop,sotheprocessruns aslong asthesystem is up.

4.4.2 The uptime timeout routine

Theroutinecountsthenumber of minutes:

void up_timer(caddr_t dummy){

mins++;if (mins == 60){

hours++;mins = 0;

}if (0 == mins % 15){

printf("%d:%02d\n", hours, mins);}

}

andprints a messageevery 15 minutes.

20

4 TUTORIAL III: PERF 4.4 TheSourceCode

4.4.3 Data for the uptime process

Theuptimepartof theprocessis completedby thetwo datadefinitions:

static int hours = 0;static int mins = 0;

andtheinclusionof theheader files for thelibrary andmessage interfaces:

#include <Messages.h>#include <stdio.h>#include <timerlib.h>

Thecontentsof theMessages.hfile producedby RTBis describedabove.

4.4.4 The Perf main process

This routine shows how to construct a low-level ROME message. The routine usesa variable, state, toidentify which measurementis beingmade,andusesthis variable to selecttheappropriate messagecode:

void perf_process(void){

ROME_MESSAGE m;ROME_PROCESS *id = rome_find_queue("resp");

timeout(perf_timer, (caddr_t)0, 10*HZ);

Thelogic for establishing andhandling thetimeoutis thesameasfor theuptimeprocess,theonly differ-encebeing thata timer is requestedevery 10 seconds.

m.dest = id;m.src = rome_this_ptr;m.opcode = (state == 0 ? ROME_M_TESTQ : ROME_M_TESTCX);m.priority = rome_this_ptr->prio;rome_send_message(&m);

The main focusof this codeis low-level messaging. The processusesthe rome_find_queueroutine tolocate theresponderprocess,andsavesthereturn value in the id variable. In orderto prepare a message,thecorerequiresthatthreefieldsbesetin themessage,theseare:

(a) the destfield, which identifies the destination processfor this message(in this casethe idvalue).

(b) thesrc field, which identifies the originator of the message. Here,the processuses the coreglobal variable rome_this_ptr, which always containsthe identifier of the currently-runningprocess in thesystem.

(c) the priority field, which specifies the scheduling priority of this message, relative to othermessages in the system. Theconvention is for a processto generatemessages at its currentscheduling priority, which is foundat therome_this_ptr->prio location.

21

4.4 TheSourceCode 4 TUTORIAL III: PERF

onefurther field is requiredin all messages,though it is not usedin thecore:

(d) theopcodefield identifies the function of this message. Note that originatedmessages have_M_ codes, thereply hasthecorresponding _R_code.

With thesefieldsset,themessagecanbesentto thedestination processusing rome_send_message.

while (1==1){

ROME_MESSAGE *mp = rome_await_message(0, 0);

switch (mp->opcode){

in this process,thereply canbeto oneof three possible messages:eitherthetimer message; or oneof thetwo perf messages.

case ROME_R_TIMEOUT :timer_tmhandler(mp);timeout(perf_timer, (caddr_t)0, 10*HZ);break;

timer replies arehandledin exactly thesameway asin uptimebut calling a differenthandler routine.

case ROME_R_TESTQ :case ROME_R_TESTCX :

mp->dest = id;mp->src = rome_this_ptr;mp->opcode = (state == 0 ? ROME_M_TESTQ : ROME_M_TESTCX);rome_send_message(mp);break;

default :rome_fatal("Bad message");

}}

}

ThePerf messagesexecute in a continuous loop, whenthe reply is received the message fields areresetandthemessageis sentoff again, possibly with adifferent operationcode, dependingonthestatevariable.

4.4.5 The respQueueHandler

All messages sentto therespprocessfirst go throughthequeue handler:

int resp_qhandler(ROME_MESSAGE *m){

if (m->opcode == ROME_M_TESTQ){

qh++;rome_auto_reply(m);

}return ROME_NOT_HANDLED;

}

22

4 TUTORIAL III: PERF 4.4 TheSourceCode

The routine filters out the TESTQmessage, usingthe rome_auto_reply macroto sendit backto its des-tination. This macroinverts the src anddestfields in the message, changesthe opcode to the _R_ formandreturnsthemessagebackto thecoreindicatingthat thecoreshould schedule themessage on thenewqueue.

4.4.6 The respmain process

Therespondermainprocessusesthesamestatevariable asPerfto determinewhichtestto run. Test0 is theQueueHandlertest(which never appears hereat all), test1 is thebasicmessage-handling measurement,test2 is theprocedurecall test,andtest3 is theprotectedprocedurecall test.

void resp_process(void){

while (1==1){

ROME_MESSAGE *m = rome_await_message(0, 0);

ph++;if (state == 2){

resp_proccall(m);}else if (state == 3){

resp_iproccall(m);}rome_reply(m);

}}

rome_reply is theequivalentmacroto rome_auto_reply for usein code otherthanQueueHandlers.

4.4.7 The respprocedures

Theresponderpartof thecodeis completedwith thetwo routinesto measure procedurecall times:

void resp_proccall(ROME_MESSAGE *m){

ch++;}void resp_iproccall(ROME_MESSAGE *m){

int old = rome_start_critical();

ih++;rome_end_critical(old);

}

Although themparameter is not usedin eitherof theseroutines,theoverheadof passing a parameter intoa routine givesa better measure of the true costof a routine. The resp_iproccall routine usesthe coreroutinesto establish a critical section around thecounter update.

23

4.4 TheSourceCode 4 TUTORIAL III: PERF

4.4.8 The perf timeout routine

The timeout routine which runs as part of the perf processevaluates the performancecalculations andupdatesthestatevariable from testto test.

void perf_timer(caddr_t dummy){

if (state == 0){

qh = qh / 10;qh_t = 100000000 / qh;printf("Queue-Handling %d.%02d\n", (qh_t / 100), (qh_t % 100));qh = 0;state = 1;

}else if (state == 1). . .

}

The appropriate counter variable is divided by 10 to yield the numberof operations per second. This isconvertedinto micro-secondsandhundredths of micro-seconds,which arethenprinted.

4.4.9 Variables

Finally, theexecutablepartof thesourcefile containsdefinitionsfor thevariablesshared betweentheperfandrespprocesses.

static int state = 0;static int qh = 0;static int ph = 0;static int ch = 0;static int ih = 0;static int qh_t = 0;static int ph_t = 0;static int ch_t = 0;static int ih_t = 0;

Sincethecountervariablesaresharedbetween two processesa natural question maybewhy two of them(ph andch) areupdatedoutsidecritical sections. Theusual requirementis indeedthataninterlock wouldberequired. However, becausethetwo processesexchangemessagesat thesamepriority, neither is pre-emptible by theother, soeachprocessrunsto completion before the otheris scheduledandno interlockis required.

4.4.10 The RTB entry

The only unusual aspectof this moduleis that the RTB entry containsthreeprocess lines, onefor eachprocessin thefile. Therespprocessregisters its QueueHandler asthethird of thestandardprocessentrypoints.

Thisemphasisesthepoint thattheremaybemorethanoneprocessassociatedwith amodule,evenwithina single source file. Eachprocessrequiresthecorresponding entry in theRTB modulecontrol file.

24

5 TUTORIAL IV: TIMER_PC 4.5 Running theimage

4.5 Running the image

Theimagecanbecompiledandinstalled asusual. Whenit is bootedandstarted,theimageentersa loopprinting thefour linesof performancemetrics,forever:

ROME Initialising.Copyright 1997 NEC USA Inc.Built by leslie on pc-rome.ccrl.nj.nec.com at Tue Apr 3 18:45:24 2001Starting the Scheduler.....perf: Queue-Handling 0.31perf: Context Switch 1.29perf: Routine Call 0.01perf: Protected Routine Call 0.08perf: Queue-Handling 0.31perf: Context Switch 1.29perf: Routine Call 0.01. . .

andevery 15 minutestheuptimeprocessrunsandproducesanextra message:

uptime: 0:15

The exact performancefigures vary with the exact modelof CPUon which Perf is running andthe be-haviour of theinstructioncache.

Understanding the impact of theselow-level timings is not trivial. The results themselvesmay vary ac-cording to the line cachealgorithmsandtheexact locationsat which instructionsareassembled. ROMEgivessomecontrol over this by allowing external control over the order in which modules areincludedin the system, and the start addressof the text. A full analysis of caching and code-placement is be-yond thescopeof this tutorial. Themain insight to begainedfrom theperformancemeasurementsis forcross-comparisonsbetween differentboardsfor potential applications.

It maybepossible to tradeoff implementationsfor particularachitectures;for exampleto removeaQueueHandler on a systemwhereroutine callshave a high overhead. In theorythis couldbeachievedthroughmoduleconfiguration andoption directives,but this hasnot yet beenexplored for version 1.0of ROME.

5 Tutorial IV: timer_pc

This tutorial introducesthefacilities in ROME for handling devices,using thecounter/timermodulesforthe Intel system. This alsoservesto ‘close the loop’ on thepreviousexampleby showing theprocessingin thetimer shared library.

In many operatingsystems,thecode for handling devices runsin a very special environment which addsto the complexity of producing a reliable implementation. In ROME, a device driver is just a module,with a processandpossibly a queuehandler, asdescribedabove. Therearetwo requirementsplaced ondevice driversthat fall outsidethescope of the interfacesdescribedso far. This tutorial introducesthesetwo concepts.

Thefirst is therequirementto run code associatedwith initialising thedevice. Usuallythis codemustrunbefore thenormalsystemscheduleris started, to setup theregistersandlow-level operation of thedevice.

25

5.1 TimerFunctionality 5 TUTORIAL IV: TIMER_PC

In ROME this is the other routine name(the init routine) that canbe given on the RTB processdialog.Theinit routineis calledfrom thecoreduring system startup before thescheduler runs. Theroutine runswith interruptsdisabled,andcannot usethemessage-passing routines.

Thesecond requirementis to install a routine for handling device interrupts. This is doneby calling therome_add_handler coreroutine,giving the interrupt numberandtheroutine to beused.Usually this callis placedin the init routine, so that the handler is in placewhenthe scheduler is started, at which pointinterruptsareenabled.

5.1 Timer Functionality

Most PC architectures implement a standard built-in counter/timer. A special-purposeregister may beloaded with a timer valueto causean interrupt at a later time. The timer initialisation routine usesthisregister to setup aninterrupt every millisecond. Themodulealsomaintains a queue of timeout requests,linked in timeout order. At each‘tick’ thecounter at theheadof queue is decrementedby one. Whenthecounter reacheszero,thereply is sent. Theoperationof adding a message to thequeueis handled withinthetimer processQueueHandlerandthereply is sentfrom theInterrupt Handler.

The user interfaceto the timer processhides the details of the message format behind the timeout andtimer_tmhandler routines, as usedin the previous tutorial. Theseroutines are provided as part of theshared-library enviroment,like the ‘C’ runtimes. The usualpractice in ROME systemsis to provide aroutineto construct a message on behalf of a caller. This alsoextendsto thedataflow messagesdescribedin thefollowing chapters.

Themodulealsousesa setof Optionfields for testing thetiming code.Sincethis is not the focusof thistutorial, thetesting codeis not includedin thelistingshere, but is available in thefull sources.

5.2 The Target File

Thevaluethatmustbeprogrammedin to thePCtimer registerdependson theexactprocessorimplemen-tation andthespecific hardwareconfigurationof thetarget system.Onepossible meansof specifying thissystem-dependentvaluewouldbeasanOptionline in RTB. However, suchvaluesdonot (usually) changebetweenapplicationsbuilt for thesamehardware.All thevarioushardwaredependenciesfor asystemareinsteadgathered togetherinto asingledescription for thathardware. Thesearethefiles thatarereferencedby theTarget component in thebuild files.

The target file contains the restof the informationneeded to complete the full configuration picture fortheexamplespresentedso far. It containsthe informationused to setup theMakefile rules, to selectthecorrect cross-compilerandassembler andto generatethe appropriate ld output format. It alsocontainsdatawhich appear in theHardware.h file in the includedirectory.

Within themastertreeis theTargetsdirectorywhichcontainsthesupported systemconfigurations. In hereis theSPB450file which describesthePCenvironment usedin this tutorial. Within thesystemfile arethedefinitionsfor this configuration:

#define TIMER_TICKS2SEC 1000#define HZ KC_TICKS2SEC#define TIMER_TIX 1193#define TIMER_VEC_TICK ICU_IRQ0#define TIMER_CLEAR_INT CPU_IOWR1(0x20, 0x20)

This alsodefines theHZ variable usedto convert from secondsto clock ticks at the timeout interface.

26

5 TUTORIAL IV: TIMER_PC 5.3 TheRTB File

5.3 The RTB File

The timer_pc modulecomprisestwo source files, onefor the low-level driver andonefor the interfacelibrary. The interfaceis specified through an external header file. In the ‘C Files’ part of the projectare two source files, timer.c and timerlib.c. In the ‘H Files’ part is timerlib.h, which is alsomarked as‘exported’ sothat it appearsin theproject’s includedirectory.

TheTIMEOUT inter-processmessageis defined in theTimerMessageSetwith asingle integer parameter:

Message TIMEOUT{

int ticks;}

Theprocessdescription containsentriesfor theinitialisation routineandqueuehandleraswell asthemainprocess.

5.4 The timerlib shared library

Thepurposeof the timerlib library is to hide the details of the timer message interfacefrom applicationprogramsandto implement the ‘callout’ functions required in many applications. The implementationof the timeoutroutine is lessstraightforward thanmight appear becauseof the needto storethe contextinformation(thepointerto theroutineandits argument)to allow thereplyto becorrelatedwith theoriginalrequest.

5.4.1 Local Data

Thelibrary usesa privatedatastructureto storecontext information:

typedef struct{

ROME_MESSAGE m; message sent to timerint flag; if still activevoid(*func)(caddr_t); completion routinecaddr_t funcarg; argument to routine

} TMOUT;

Thestructurecontains theactual instanceof theROME_MESSAGE which is sentto thetimer process.

5.4.2 The timeout routine

The timeoutroutine usedabove is actually a macrointerfaceto the timeout_proc routine. This routineallocatesa TMOUT structure,initialisesthefieldsandsends themessageto thetimer process.

int timeout_proc(ROME_PROCESS *who, void (*func)(caddr_t),caddr_t se, int hz)

{TMOUT *tm = (TMOUT *)rome_alloc(sizeof(TMOUT), ’t’, 1);ROME_T_TIMEOUT *to = RCAST(ROME_T_TIMEOUT, tm->m);

27

5.4 Thetimerlib sharedlibrary 5 TUTORIAL IV: TIMER_PC

Thefirst line of the routine usesthecorerome_alloc routine to allocateandcleara block of memoryforthe context. rome_alloc halts the system if no memoryis available, so the routine herehasno needtocheckfor NULL pointers. Thesecond parameteris anallocation identifier, which canbedisplayedfromthedebugger, indicatingfrom wherethis allocation requestoriginated.

Thesecond line introducesa new macrofrom theROME core. TheRCASTmacrotakesa message typeasits first argument,andaROME_MESSAGEstructureasits second argumentandreturnsapointer to theargumentfields of a message, castinto the correct type for the message. The RCASTPmacroperformsexactly thesamefunction except that its second argumentis a pointerto a ROME_MESSAGE structure.

tm->flag = 1;tm->func = func;tm->funcarg = se;to->ticks = hz;tm->m.src_context = tm;tm->m.dest = rome_find_queue("timer");tm->m.src = who;tm->m.opcode = ROME_M_TIMEOUT;tm->m.priority = who->prio;rome_send_message(&tm->m);return (int)tm;

}

By default, processesgenerate messages at their current scheduling priority. The message priority isinitialisedfrom theprocess’current state.This allowsthetimer library to run asa sharedlibrary possiblywith concurrent execution in multiple contexts. Themacrois actually:

#define timeout(a, b, c) timeout_proc(rome_this_ptr, a, b, c)

TheROME coremaintains the rome_this_ptr variable andtheappropriate fieldsareavailable ascontextsareswitched. Thereasonfor the timeoutandtimeout_proc level of indirection is to allow certain routinesto bypassthe current-processpointer, for exampleto create timerson behalf of other processes. This isneeded,for example,to createa timer within aninterrupt handler. All of thevariablesfor theroutineareheldon theprocessstack, sotheroutine is truly sharable. No interlocksareneeded for any of this routine.

The hz argument is usedto initi alise the ticks parameterof the message, which is sent to the “timer”process. For this routine to function correctly, the timer mustbe linked into the system with the name“timer” , but the operation of the library interfacedoesnot depend on the implementation of the actualcodeusedfor thesystemtimer.

ROME messages contain fields that canbe usedby the sender andthe recipient of the messageto storeinformation. Here,the src_context field is used(since this is the source of the message), to point to thetimeoutstructure.

5.4.3 The untimeout routine

The untimeout routine is usedto cancel a timer. This is lessobvious than it might appear, asthe timermessage may be in a numberof places: it may still be on the queue of the timer process,or it may bequeued back to the originator. The ROME implementation takes the ‘easy way out’, by allowing thetimeoutto occur, but not calling theuser’s routine. It signalsthis by setting theflag field to zero:

28

5 TUTORIAL IV: TIMER_PC 5.4 Thetimerlib shared library

void untimeout(int cx){

((TMOUT *)cx)->flag = 0;}

This alsoallows theallocatedmemoryto befreedin a cleanway.

5.4.4 Sleep

Thesleeproutinesuspendsa processfor a numberof timer intervals,using a timeout message. That is, itprevents theprocessfrom accepting any messages,or otherwiseoccupying the CPU,during that period.In order to do that,it mustwait until thereply to thetimeout comesbackto theprocess:

void sleep(int hz)

{ROME_MESSAGE m;ROME_T_TIMEOUT *to = RCAST(ROME_T_TIMEOUT, m);

to->ticks = hz;m.dest = rome_find_queue("timer");m.src = rome_this_ptr;m.opcode = ROME_M_TIMEOUT;m.priority = rome_this_ptr->prio;

rome_send_message(&m);rome_await_message(&m, 0);

}

Sinceit only makessenseto sleepin a processcontext, the rome_this_ptr variable canbe useddirectly.Mostof thiscodeis similar to theinternal timeout_proc routine,exceptthatnoTMOUT context is needed.Themaindifferenceis thefinal line, which is anexplicit call to rome_await_message. By giving apointerto a message asthefirst parameterto rome_await_message, thecoreschedulersuspendstheprocessuntilthat particular messageis placedon the process’ messagequeue. That message is thenreturnedastheresult of thatparticular call, irrespective of any othermessageswhich maybeon thequeue. Because themessage is local to thesleeproutine, it canbeallocatedon theprocess’local stack.

Theapproachof usinglocal messages andwaiting for the replies is usedin the interfacelibrary routinesto themaindataflow messages,explainedbelow. In this way, it is possible to mix event-drivenmessageswith blocking calls.

5.4.5 The timer_tmhandler routine

The timer_tmhandler codecompletesthecycle by calling theuser’s routine andreleasingtheresources:

void timer_tmhandler(ROME_MESSAGE *m){

TMOUT *tm = (TMOUT *)m->src_context;if (tm->flag){

(tm->func)(tm->funcarg);}rome_free(tm);

}

29

5.5 TheClock routines 5 TUTORIAL IV: TIMER_PC

The source context field in the messageis usedto recover the pointer to the TMOUT structure,andtheflag field is checkedbefore theuser’s routine is called. Finally, rome_free, which is a coreroutine,returnsthecontext memoryto the(global) freememorypool.

5.5 The Clock routines

Theother half of thetimer support is thesetof routinesto interfacewith thehardware.

5.5.1 The timer init routine

Thefirst routine to becalledis theinitialisation routine:

void timer_init(void){

tick_counter = 0;timerq = NULL;rome_add_handler(TIMER_VEC_TICK, timer_isr);

CPU_IOWR1(TIMER_CONTROL, 0x36);CPU_IOWR1(TIMER_STATUS0, TIMER_TIX & 255);CPU_IOWR1(TIMER_STATUS0, TIMER_TIX >> 8);

}

The routine initialises the variables usedby the interrupt routine and registers the handler for the ticktimer usingthe coreroutine rome_add_handler. The interrupt vector numbercomesfrom the hardwaredescription in thetarget file. Theroutine alsosetsthevalues for thetimer registers. Accessesto machine-dependentarchitectural features(like IOSpace)arehandled by theCPUplug-in through macros. In thiscase,CPU_IOWR1writesa single byte to a particular IO location. More details of these macrosis givenin theCPUplug-in documentation,andtheROME Porting Guide

The timerq variable is used to hold thequeueof timer requestsasa linkedlist of messages.

5.5.2 The QueueHandler

This routine controls thequeue of timer requests:

int timer_qhandler(ROME_MESSAGE *mptr){

switch (mptr->opcode){case ROME_M_TIMEOUT :{

ROME_T_TIMEOUT *to = RCASTP(ROME_T_TIMEOUT, m);

mptr->link = 0;if (to->ticks <= 0){

rome_auto_reply(mptr);}

30

5 TUTORIAL IV: TIMER_PC 5.5 TheClock routines

This first check filters out any messageswith timeoutrequestsin thepast,or with zerotimeoutvalues (asmight ariseif theinterval is calculatedby theoriginatorbasedon processing activity).

if (timerq){

ROME_MESSAGE *mp = timerq;ROME_MESSAGE *pp = timerq;ROME_T_TIMEOUT *tp = RCASTP(ROME_T_TIMEOUT, mp);

while (tp->ticks < to->ticks){

to->ticks -= tp->ticks;if (mp->link){

pp = mp;mp = mp->link;tp = RCASTP(ROME_T_TIMEOUT, mp);

}else{

mp->link = mptr;return ROME_HANDLED;

}}

Timeout messages are chained in a linked list in timeout order, with the ticks value representing thedifferencebetween theprevioustimer andthecurrent one.This meansthat theshortesttimeoutis alwaysat theheadof thequeue,alsothatmultiple timersfor thesametick will appearwith zeroticks in thechain.Sincelinkingin messagechains is acommonoperation (alsoused within thecore), eachmessagecontainsan explicit link field intendedto point to another message. The first loop scansdown the queue for theright place to put thenew message. Themessage is addedto theendof thequeue if thereis no messagewith a longertimeout interval.

if (mp == timerq){

mptr->link = timerq;timerq = mptr;

}else{

pp->link = mptr;mptr->link = mp;

}tp->ticks -= to->ticks;

}else{

timerq = mptr;}return ROME_HANDLED;

}break;

Thefirst testhandlesadding thenew messageto theheadof the timer queue,or adding to themiddleofthequeue (inbetweenpp andpp->link). Thefinal elseclausehandlesthecaseof anemptyinitial queue.

31

5.5 TheClock routines 5 TUTORIAL IV: TIMER_PC

In all cases the result returned is ROME_HANDLED to indicate to the core that the messagehasbeenprocessedin thequeuehandler

}return ROME_NOT_HANDLED;

}

Finally, all othermessagesarereturnedwith the‘not handled’ indication, meaningthey will bepassedonto themainprocess(below).

5.5.3 The Main Process

Themainprocess codeis very simple:

void timer_process(void){

while (1==1){

ROME_MESSAGE *m = rome_await_message(0, 0);

rome_kprintf("timer message %x??\n", m);}

}

The only messagethat is supposedto be sent to the timer processis the TIMEOUT message. So allmessagesare,in theory, handledby theQueueHandler, andthemainprocessshould neverbewokenfromits wait. Themainpurposeof theprocess is to provideareferencefor the“timer” processname.Theprintservesto alerta userto a probablemisconfiguration of thesystem, asit should never appear.

5.5.4 The interrupt handler

This routine is called from within theCPUplug-in in responseto thetimer interrupt:

void timer_isr(int ino){

if (timerq){

ROME_T_TIMEOUT *tp = RCASTP(ROME_T_TIMEOUT, timerq);

tp->ticks--;while (timerq && tp->ticks <= 0){

ROME_MESSAGE *head = timerq;timerq = head->link;rome_reply(head);tp = RCASTP(ROME_T_TIMEOUT, timerq);

}}if (++tick_counter == KC_TICKS2SEC){

tick_counter = 0;

32

5 TUTORIAL IV: TIMER_PC 5.6 Modifying theClock Driver

secs++;}TIMER_CLEAR_INT;

}

If thereis a timeout messageon thequeue,thecountremaining for thatmessageis decremented.Whenitreacheszero, all messagesfor thattick arereturnedasreplies.Whentheinternal tick_counter hascountedfor onesecond it is resetandthe internal secscounter is incremented.Finally, theTIMER_CLEAR_INTmacro,definedin the target file, is usedto clear the pending timer interrupt. Note that both the QueueHandler andtheInterruptHandleraremanipulatingthetimerq variable. It is essential thatneitherinterfereswith the operation of the other. In this case,the automatic protection offered to the QueueHandlerofrunning as a critical section is usedto ensure that the interrupt handler cannot run during the update.InterruptHandlersalsorunascriticalsections,soresourceinterlocking is achievedwithout further explicitprocedurecalls.

5.5.5 Variables

static int secs = 0;static int tick_counter;ROME_MESSAGE *timerq;

The timer queuevariable is local to this file, but exporting the symbol helpswhen debugging timingproblems(seealsothedebug partof chapter7).

5.6 Modifying the Clock Dri ver

The previoussections have described the functionality of the clock processasusedin the previoustwotutorials. No new builds areneeded for this description.

The second purposeof this tutorial is to modify the clock driver to print a ‘+’ sign every second. Thiswill beproducedfrom theclock interrupt handler, which runsin anenvironmentwhereprintf cannot beused. ROME providesa special routine to generateoutput in theseenvironments, called rome_kprintf.Theroutine runs asa critical section andwritesdirectly to theUART.

5.6.1 Making the source file writable

If thesourcesfor moduleshave beencheckedout from a CVS repository, thena modulemustbe lockedbefore it canbeupdated. This is donein RTBby selecting themodule in theprojectsummarylisting andselecting the‘lock’ option from thepopup menu(right-click on themouse). Oncethis is done, thesourcefiles in theModulesdirectory will bewritable. It is not necessaryto rebuild theprojectMakefile tree.

5.6.2 Changing the source

Using the Perf project, lock the timer module andedit the file timer_pc/timer.c changing the followinglinesin the timer_isr routinefrom:

33

6 TUTORIAL V: UART16550

if (++tick_counter == KC_TICKS2SEC){

tick_counter = 0;secs++;

}

to:

if (++tick_counter == KC_TICKS2SEC){

rome_kprintf("+");tick_counter = 0;secs++;

}

thensave thechangesandrun make install asbefore.

5.6.3 Running the image

Whentheimageis run, theperformancevaluesareprintedwith ten‘+’ signsbetweenthem.

++++++++++perf: Queue-Handling 0.31++++++++++perf: Context Switch 1.29++++++++++perf: Routine Call 0.01++++++++++perf: Protected Routine Call 0.08++++++++++perf: Queue-Handling 0.31

dependingontheexactsystem timing, the‘+’ charactersmayappearin themiddleof theusual perfoutput,sincethe rome_kprintf routine runs at absolute priority and bypassesthe UART queueing mechanismwhich prevents interleavedoutput. Theimplementationof this mechanismis describedin tutorial V.

6 Tutorial V: UART16550

The main focus of this tutorial is to introducethe standardROME dataflow messages anddemonstratehow they arehandledwithin a device.

The applications so far (such asecho) have been unusual for an embeddedprocessor application in thatthedataflows terminate within themachine. A moreusualmodelwould have data beingreceivedby onedevice, beingmanipulatedby oneor moreprocesseswithin themachine andfinally leaving themachinethrough another (or eventhesame)device. If every devicedriver definedits own setof messagesfor datatransfer therewould needto bea

�����translationoperations, possibly requiring datacopies,andthere

would beno standardinterworking between modules.

Instead,ROME definesa standardsetof messages to beusedfor datamovement.Thedesign is intendedto addressa numberof issues:

(a) Thegoal is to move datafrom input device to output device without any unnecessary copies(ideally without any copies).

34

6 TUTORIAL V: UART16550 6.1 mblks

(b) For someinput devices,notably networked systems, the target of a datablock cannot (maynot) beknownuntil theblock hasbeenreceivedinto memory.

(c) Somedevicesmayrequirebuffersto beallocatedin aspecial way, for examplewithin agivenregion of memory, or not spanning particular addressingboundaries.

(d) In contrast, somedevices, for exampledisk systems,mayoperatein a completelydetermin-istic manner. Not only may it not matterwherebuffers arelocated, but the destination of abuffer canbedeterminedbeforea requestis issued.

(e) Network protocolsrequire theaddition andremoval of headers (andtrailers)from datapack-ets.

The approachusedin ROME is basedaround the ideas of STREAMS (See,for examplethe SunOS5.3STREAMSProgrammer’s Guide, (SunMicrosystemsInc., 1993)), but intendedto beusedthroughoutthesystem, not just within the‘kernel’.

6.1 mblks

An mblk, or message block, is the basic representation of datain transit through the system. An areaofmemoryusedasa databuffer is representedby three variables:

uchar *b_base; original base of bufferuchar *b_lim; absolute end of bufferuint b_type; buffer type;

whereb_base points to thestartof thebuffer andb_limpoints to theendof thebuffer. Theb_typefield isusedin protocol processingto identify thetypeof datain thebuffer.

The sizeof the buffer is b_lim-b_basebytes. Note that these areonly pointers to a buffer, the structuredoesnot itself reserve thatarea.It alsodoes not definehow theareawithin thebuffer is used.

Themblkstructurerepresentsdatawithin a buffer asfollows:

uchar *b_rptr; current read pointeruchar *b_wptr; current write pointer

The b_rptr and b_wptr fields definethe areaof the datablock containing valid data. b_rptr, the readpointer, definesthe point at which a processshould start reading data(i.e. the beginning of the buffer).b_wptr, the write pointer, definesthe point at which a processshould appenddata(i.e. the endof thebuffer). Thevalid datalie between thesetwo pointersandareof lengthb_wptr-b_rptr bytes.

Thereadpointer neednot coincidewith thestartof thedatablock, andthewrite pointer neednot point totheendof thedatablock. By setting thereadandwrite pointersfor a new messageblock someway intothedatablock it is possible to ‘reserve’ space at thestart of a block. This aspect will be thefocus of thenext tutorial.

A single unit of datamay spanmorethanonebuffer. Multiple mblks canbe chained together usingthecontinuation field, b_cont:

35

6.1 mblks 6 TUTORIAL V: UART16550

struct mblk_t *b_cont;

To completethemblk datadefinition, two fieldsareavailable for per-buffer ‘immediate’ data:

uint b_immed;uint b_immed1;

An exampleof theuseof an immediatedatafield will begivenbelow. Thebasicapproachto thesedatastructuresis thesameastheexamplesin thetimer andperformancecode.Thesameportion of a ROMEmessagethatis usedto containtheticksvaluefor theTIMEOUT messagecanalsobeusedto containthesemblkfields.Thevaluescanthenbemanipulatedasthemessagepassesthroughthesystem. However, sincemostmessagesusethemblkfields,rather thanrequiring RCASTor RCASTPmacrosto access thefields,they canbe accessed directly within the message. In fact, to simplify the porting of STREAMS-baseddriversto theROME environment,theROME_MESSAGEandmblk_t datatypesareoneandthesame!

This hasgivenenough background to allow thefull ROME_MESSAGE structureto begivenasit appearsin rome.h. The useof the dest, source, opcode, priority, dest_context andsrc_context fields have beencovered in the previous tutorials. The link field was usedin the Clock driver, but is also usedwithinthe corefor chaining messagesto processqueues. Eachmessagealsocontainsan error codefield. Theremainder of thefieldsareusedto contain anmblk:

#define b_cont residue.imblk.ib_cont#define b_rptr residue.imblk.ib_rptr#define b_wptr residue.imblk.ib_wptr#define b_base residue.imblk.db_base#define b_lim residue.imblk.db_lim#define b_type residue.imblk.db_type#define b_immed residue.imblk.immed#define b_immed1 residue.imblk.immed1#define m_args residue.args.avec

#define RCAST(_t, _m) (_t *)((_m).m_args)#define RCASTP(_t, _m) (_t *)((_m)->m_args)

typedef struct _rome_message{

struct _rome_message *link; to next on queuestruct _rome_process *dest; destination queuestruct _rome_process *src; source queueint m_errno; see errno.huint opcode; see messages.huint priority; this message priorityptr dest_context; opaque field for dest useptr src_context; opaque field for src useunion{

struct{

struct _rome_message *ib_cont; continuation blockuchar *ib_rptr; current read pointeruchar *ib_wptr; current write pointeruchar *db_base; original base of bufferuchar *db_lim; absolute end of bufferuint db_type; buffer type

36

6 TUTORIAL V: UART16550 6.2 Themessage interface

uint immed; immediate datauint immed1; immediate data 1

}imblk;struct{

uint avec[8]; depends on code}args;

}residue;}ROME_MESSAGE, mblk_t;

Thestructurecontainsthe linkage fieldsandcontext fields followed by a 32-byte argumentarearesidue,which canbe viewed either asan 8-word array args or as imblk. The conventional namesfor the mblkfieldsaredefined symbols addressingthese fields.

6.2 The messageinterface

This section describesthemessagesthataredefinedaspartof thestandardROME interface.Threemes-sages, OPEN,CLOSEandFLUSH, handle control operations. Six messages,FETMBLK, GETMBLK,NEWMBLK, OUTMBLK, PUTMBLK andRETMBLK areusedfor moving buffersof dataaround thesystem.

6.2.1 Control Messages

ROME_M_OPEN

In orderto useROME dataflows between processes,thedestination processmustbelocatedandany statewithin that processestablished for the particular flow. Locatingprocesses, aswith the timer library, isdonewith thecoreroutinerome_find_process. Initialising thestatein thedestination is doneby sendinganOPENmessageto thedestination. Theremaybemany separateflowsbetweenany two pairsof processes,eachwith its own internal state,andthereneedsto besomewayof associatingmessageswith flows. Whena processreceivesanOPEN message it hastheoption of storing a ‘context’ valuein thereply to identifytheflow. This 32bit valueis returnedin thedest_context field of thereply. Thesamevalue will besetonall subsequent messageson thatflow. This is handledby theruntime library which implements theFILEabstractionon top of thebasic messages.

TheOPENmessagehasits own parameter area:

typedef struct{

struct _rome_url openurl;int mode;

}ROME_T_OPEN;

Thefirst parameteris apointerto aROME URL datastructureusedto identify thedestination, thesecondis a bitfield of open types. Currently this bitfield contains the BINARY, READandWRITEbits derivedfrom the‘C’ fopen routine.

The full openurl is an expandedform of the Uniform Resource Locator (URL) wherethe schemepartis usedto identify a ROME process(see, for example ‘Uniform Resource Locators’, RFC 1738). TheROME URL will bedescribedin moredetail in thenext chapter.

37

6.2 Themessageinterface 6 TUTORIAL V: UART16550

ROME_M_CLOSE

The CLOSEmessage is usedto inform a process that it can terminate a flow and release all the flowresources.Theonly information passedon theCLOSEmessageis thedest_context field.

ROME_M_FLUSH

Any processin thesystem is allowedto buffer data.It mayevenreturn a reply to theoriginatorindicatingthat the datahasbeen‘handled’ before a transactionhascompleted(for exampleTCPmayreply beforethe otherside hasacknowledgedthe data). The FLUSH message synchronisesa dataflow. The reply toa FLUSH should not be generated until all previous datahave been transmittedandacknowledged(forreliable service).

6.2.2 Moving Data with Messages

Having established thecontext for a flow, datacanbemovedalongit. Themainissue is themanagementof the memorycontaining these data,asa system resource. That is, who allocatesthis memory;who iscurrently ‘responsible’ for it; whohastheability, or therequirement, to freeit whenit is nolongerneeded;andwho determinesthatthedataareno longer needed?

In the context of ROME, these arenot trivial questions. The dataflow mechanism is designed to allowbuffers, representedby mblks,to bepassed betweenprocesses,andthegoal is a truezero-copy architec-ture. This implies that theprocessthatoriginally filled a buffer is unlikely to be theonethat decides thedatain it areno longer needed. Theprocessthat doeswish to free the buffer may have no direct accessto, or knowledge of, the processthat created it. Also, somedevicesplace constraintson their buffers –they mayhave special alignmentor boundaryrequirements(such aslying within a 64k addressing page)or they maylive on special memory(for exampleuncachedmemoryfor DMA-capable devices).

For somedevices, the ultimatedestination process of a buffer of datamay not be known until that datais completely received(for exampleethernetframes).In contrastsomedevices,such asdisk drives, onlydeliver data in responseto explicit requestsandofteninto buffers associatedwith thoserequests.

TheROME dataflow modelattempts to solve these issuesin assimplea way aspossible, usingonly sixdatamovementmessages.

ROME_M_FETM BLK

TheFETMBLK message is a requestto fill an mblk with data. Thebuffer for the datais supplied at thetime to message is issued(i.e. it is ownedby the originator). Therecipient should fill in datastarting attheb_wptrvalueandupdatethatfield to reflectthenew length. TheFETMBLK messageis typically sentto a processthatdoesdeterministic datamovement,for examplea disk device.

ROME_M_GETMB LK

The GETMBLK messageis a request for a block of data. The buffer for the data is allocated by therecipient, andmustbe returned to it. The reply contains the mblk fields initialised to point to the databuffer. Note thata single buffer of datamayspanmultiple blocks, these arechainedthrough the b_contfield. Thebuffer is returnedto theownerwith theRETMBLK message.

ROME_M_NEWMB LK

The NEWMBLK messagerequestsan emptybuffer into which the sender canwrite data. The length ofthebuffer is passed in theb_wptrfield of theoriginal message. Thereply will have theb_wptrfield setto

38

6 TUTORIAL V: UART16550 6.3 Application to theUART16550driver

point to thefirst writablebyteof thedatabuffer. Note thatprotocol header processing requirementsmaymeanthat this is not theb_baselocation. Thebuffer mustbereturnedto theallocatorwith a PUTMBLKmessage.

ROME_M_OUTMB LK

TheOUTMBLK messagerequeststherecipient to transmit theassociated data block in the‘downstream’direction (towardsthe outside world). When transmission is complete, the databuffer is returned, un-changedto theoriginator, i.e. it is a ‘write andkeep’ operation.

ROME_M_PUTM BLK

ThePUTMBLK message requeststherecipient to transmit theassociateddatablock in the‘downstream’direction (towardsthe outside world). When transmission is complete, the databuffer is freed. Thisrequires that the recipient originally allocatedthe buffer andyielded it with a NEWMBLK message. Totransmit datathrough a process which doesnot own thebuffer, theOUTMBLK messagemustbeused.

If thedatalengthin a PUTMBLK messageis zero(i.e. b_wptr== b_rptr) no dataaretransmittedandthebuffer is freedimmediately. This maybeusedto returna buffer that is no longer required,or to return abuffer for data previously transmittedby anOUTMBLK operation.

ROME_M_RETMB LK

TheRETMBLK messageis usedto return a databuffer thatwaspreviously passedupstreamin a GETM-BLK reply. Note that RETMBLK returns buffers from GETMBLK andPUTMBLK returns buffers fromNEWMBLK. This allows differentbuffer strategiesto beimplementedfor transmission buffersandrecep-tion buffers.

6.3 Application to the UART16550dri ver

The UART16550 driver providesan interfaceto any UART implementation based on the NS16550reg-ister map. It providescharacters, one-at-a-time, to satisfy input requests, and outputs charactersfromPUTMBLK or OUTMBLK requests. The UART doesnot itself perform characterechoing, or interpretformateffectors (for examplebackspaceanddelete). It is alsoexpectedthatonly oneprocesswill openadatapathto theUART directly. In thesesystemsthat is theConsole module.

Sincethe UART is a terminating device which does not itself originatemessages,it only hasto processtherequest formsof themessages.A processwhichsits‘mid-stream’musthandlerequestsfrom upstreamandrepliesfrom downstream.This is covered in thenext tutorial.

6.4 The Target File

Because thecodeis designedto work with morethanoneimplementation, variousparametersareused inthetarget file to configurethesourcesat compilation time:

#define SERIAL_UART16550_BASE0 0x2f8#define SERIAL_UART16550_BASEX 0x3f8#define SERIAL_UART16550_VEC_INT0 ICU_IRQ4#define SERIAL_UART16550_VEC_INT1 ICU_IRQ3#define SERIAL_UART16550_REG_SPACING 1#define SERIAL_UART16550_MCR (MCR_OUT2 | MCR_DTR)

39

6.5 SourceCode 6 TUTORIAL V: UART16550

#define SERIAL_UART16550_ADD_HANDLER(_a, _b) rome_add_handler(_a, _b)#define SERIAL_UART16550_CLEAR_INT0 CPU_IOWR1(0x20, 0x20)#define SERIAL_UART16550_CLEAR_INT1 CPU_IOWR1(0x20, 0x20)

Thesedefinitionsencapsulatethevariationsthatarefoundbetweendifferentboardsthatusethis chipset.For example, in somearchitectures, the UART registersarespaced onebyte apart, which in other theyarespace four bytesapart (depending on the addressdecode logic). This is setby the REG_SPACINGparameter.

6.5 SourceCode

TheUART drivercontainsthesamebasicstructureasthat clock driverof theprevioustutorial, thoughtheoperationof theindividualroutinesarerathermorecomplicated,asmany moremessagetypesarehandled.As themain focus of this tutorial is on message handling, someof thedevice-specific code(for examplehandling multiple UARTs) will be omitted. All the device codeuses a setof macrosto encapsulatetheaccessesto theUART registers.Thesemacrosin turnusetheCPU-specificIOSpaceaccessmacrosdefinedby theplug in, for example:

#define WR(_r, _v) CPU_IOWR1(port->uart_port + (_r), (_v))#define RD(_r) CPU_IORD1(port->uart_port + (_r))

for writing andreading a byte to the_r register of thecurrent port.

Thestructure of this tutorial differs from the previous ones.Ratherthango through the QueueHandler,ProcessandInterrupt Handler routinesline-by-line, thefoll owing sectionsfocuson how eachmessageishandled within themodule.

6.5.1 The init routine

A short routineensuresa cleanstatefor therestof thecode:

void serial_init(void)

{SERIAL_PORT *port;

port = &ports[0];port->uart_port = (volatile uchar *)SERIAL_UART16550_BASE0;SERIAL_UART16550_ADD_HANDLER(SERIAL_UART16550_VEC_INT0, serial_isr);port->readq.head = port->readq.tail = NULL;port->writeq.head = port->writeq.tail = NULL;port->todo = ’\0’;WR(IER, IER_RxIE | IER_RLS);

}

The interrupt handler usesa queue of readrequestsfor input, anda queue of write requestsfor output.Input messages arereturned in the order they arereceived andoutput is sentto the UART in the orderin which it is generated. Theseboth require FIFO queues,which areaddedto at the tail endandfromwhichelementsareremovedat theheadend.This is asufficiently commonrequirementfor messagesthatROME supports this queue by a combinationof macros andshared-library routines.Theroutinesrequire

40

6 TUTORIAL V: UART16550 6.5 SourceCode

only thatthehead andtail pointersof a queue beNULL whenthequeueis empty. This is ensuredaspartof themodule initialisation:

Thefinal line of theroutineenablesthereceive interrupt for UART0.

6.5.2 The Interrupt Handler

Thereis somecodein theinterrupt handler that is not directly relatedto message handling.

static void serial_isr_internal(int uix){

SERIAL_PORT *port = &ports[uix];uchar ipend = RD(IIR);

#ifdef ROME_TRACE_INTERRUPTSrome_add_trace((ptr)(uint)ipend, ROME_TT_STARTINT, (ptr)uix);

#endif

ipend &= IIR_MASK;

if (ipend == IIR_RDA || ipend == IIR_TIMEOUT){

. . . input handling}if (ipend == IIR_THRE){

. . . output handling}

#ifdef ROME_TRACE_INTERRUPTSrome_add_trace(0, ROME_TT_ENDINT, (ptr)uix);

#endif}

Thecallsto therome_add_traceroutinerecord theentry andexit to theinterrupt handler, if theROME_TRACE_INTERRUPTSoption is set. This debugging aid is described later in this chapter. The interrupt handler readsthestatusregister of the UART which generatedthe interrupt andprocessespending input charactersandpendingoutput characters.

6.5.3 The OPEN Message

QueueHandler

ROME_T_OPEN *oopen = RCASTP(ROME_T_OPEN, mptr);uint uix = open->openurl->port;

mptr->dest_context = (ptr)uix;mptr->m_errno = 0;rome_auto_reply(mptr);

TheUART codeis unusualin handling theOPEN messagein theQueueHandler. It doesthisto preventtheConsole processfrom blocking whenit opens thedataflow to thedefault output device. Having Console

41

6.5 SourceCode 6 TUTORIAL V: UART16550

complete processstartup without blocking prevents circular messagelockup with processeswhich aretrying to open standardstreams. It is assumedthat only oneprocessever opens a file directly to eachUART port, sothereis no need to record any special per-processcontext information. Theroutinereturnstheindex of theopenUART asthesourcecontext in thereply. TheUART index is derivedfrom theURLstructures,which is explainedin thenext chapter. In all the following messages,theport index is passedin throughthemessage, andthecontext is recoveredusingthefoll owing declaration:

SERIAL_PORT *port = &ports[(int)mptr->dest_context];

this line is implied at thestartof all thefoll owing code fragments.

6.5.4 The CLOSE Message

Main Process

ROME_MESSAGE *xx = rome_remhead(&port->readq);

while (xx){

xx->m_errno = ECANCELED;rome_reply(xx);xx = rome_remhead(&port->readq);

}mptr->m_errno = 0;rome_reply(mptr);

In processingtheCLOSErequest, any messagesbeingheldby the UART for that flow mustbe returnedto theoriginatorwith anerror indication. This is handledwithin theprocess.Notethatall therepliesarequeuedto theoriginator before thereply to theCLOSEitself.

6.5.5 The FLUSH Message

Main Process

if (port->writeq.head == 0){

rome_reply(mptr);}else{

int old = rome_start_critical();

rome_addtail(&port->writeq, mptr);rome_end_critical(old);

}

Therequirementfor FLUSH is thatthemessage is returnedonly afterall preceding output hascompleted.If there is no output pending, themessage is returnedimmediately; otherwisethemessage is addedto thetail of theoutput queue. Notethatin this case,running in theprocesscontext, theroutinemusttake outaninterlock in order to updatetheport writeq variable.

Interrupt Handler

42

6 TUTORIAL V: UART16550 6.5 SourceCode

while (port->writeq.head && port->writeq.head->opcode == ROME_M_FLUSH){

mp = rome_remhead(&port->writeq);rome_reply(mp);

}

Oncea FLUSH messagearrives at the headof the output queue(becauseall earlier buffers have beentransmitted) it canbereturnedto theoriginator.

6.5.6 The FETMB LK Message

QueueHandler

rome_addtail(&port->readq, mptr);return ROME_HANDLED;

Incoming FETMBLK requestsareaddedto thetail of theport’s readqueue.

Interrupt Handler

Seethedescription for GETMBLK, below.

6.5.7 The GETMB LK Message

QueueHandler

mblk_setup(mptr, (uchar *)&mptr->b_immed, 4);rome_addtail(&port->readq, mptr);return ROME_HANDLED;

TheGETMBLK message,which expects theUART to provide thebuffer, usesinsteadtheb_immedareaof themblk, sinceit only ever returns onecharacter. Thesupport library routinemblk_setupinitialisesallthe fields of the message block for a given buffer. The handler thenadds the message to the tail of thequeue of reads for theUART.

Interrupt Handler

char data = RD[RXD];

if ((mp = rome_remhead(&port->readq))){

*mp->b_wptr++ = data;rome_reply(mp);

}

This codelies within the ‘input handling’ section of the interrupt routine. The available input characteris removed from the UART’s FIFO. If there is a requeston the readqueue, the character is passedup inthereply. Thewrite pointer is incremented,indicatingonebyte in thebuffer. Notethat it does not matterwhether this is a FETMBLK message or a GETMBLK message at this point.

43

6.5 SourceCode 6 TUTORIAL V: UART16550

6.5.8 The NEWMBL K Message

QueueHandler

OUTPUT_BUFF *op = freeq;

if (op){

freeq = op->link;}else{

op = (OUTPUT_BUFF *)rome_alloc(sizeof(OUTPUT_BUFF), 4, 1);}mblk_setup(mptr, op->input, 128);rome_auto_reply(mptr);

Although input is handled character-by-character, output may be presented in larger buffers, typicallycontaining a full line. Available output buffersareheld in a linkedlist. If thereis a buffer, it is used,elsea new oneis allocated.Themblk_setuproutine initialisesall thefields,aswith GETMBLK above.

6.5.9 The OUTMB LK Message

QueueHandler

if (port->writeq.head == NULL){

port->currtxm = mptr;port->currtxp = mptr->b_rptr;SET(IER, IER_TxIE);

}rome_addtail(&port->writeq, mptr);return ROME_HANDLED;

If theUART transmit queueis empty, theUART mustbere-started, andthe‘transmit available’ interruptenabled. Thehandler usesthecurrtxm andcurrtxp pointersto control individualcharacter transmissions.Note that, sincethe QueueHandler runs asa critical section, updating thesevariablesis automaticallyinterlocked against theinterrupt service routine. In all casesthemessage is added to theendof thewriterqueue for theport.

Interrupt Handler

Seethedescription for PUTMBLK, below.

6.5.10 The PUTMBL K Message

QueueHandler

44

6 TUTORIAL V: UART16550 6.5 SourceCode

if (mptr->b_rptr == mptr->b_wptr){

OUTPUT_BUFF *op = (OUTPUT_BUFF *)mptr->b_base;

op->link = freeq;freeq = op;rome_auto_reply(mptr);

}(the rest of the code is the same as OUTMBLK above)

For PUTMBLK, theroutine mustcheckfor thezero-length indicationusedto return unwantedbuffers. Inthis case, the buffer is added to the free queue. Sincethere is no ordering required for thesebuffers, asimplehead-linkedlist suffices. Otherwise, theprocessing is thesameasfor OUTMBLK above.

Interrupt Handler

Thefollowing codeis containedin the‘output handling’ partof theinterrupt serviceroutine.

if (port->todo){

WR(TXD, port->todo);todo = ’\0’;

}

The todovariable holdsa singlecharacterwhich is next to beoutput. This is usedto convert ‘newline’ to‘newline’ + ‘carriage-return’ for standardterminals,without changing thecontentsof thesupplied outputbuffer.

else if (port->currtxm){

if (*port->currtxp == ’\n’){

todo = ’\r’;}WR(TXD, *port->currtxp++);

if there is a character awaiting output, theselines handles the CRLF translation, and output the nextcharacterfrom thebuffer.

while (port->currtxm && port->currtxp == port->currtxm->b_wptr){

port->currtxm = port->currtxm->b_cont;if (port->currtxm){

port->currtxp = port->currtxm->b_rptr;}

}

A single output requestmay contain a linked list of buffers, eachof which is to be printed. This codemovesthe internal pointersto the next non-empty characterposition in the chain, andleavescurrtxm asNULL if thecurrentmessagehasbeencompletely transmitted.

45

6.5 SourceCode 6 TUTORIAL V: UART16550

if (port->currtxm == NULL){

ROME_MESSAGE *mp = rome_remhead(&port->writeq);

if (mp->opcode == ROME_M_PUTMBLK){

OUTPUT_BUFF *ob = (OUTPUT_BUFF *)mp->b_base;

ob->link = freeq;freeq = ob;

}rome_reply(mp);

PUTMBLK requestscausethebuffer to befreedwhenall of thedatahave beentransmitted,sotheoutputbuffer is linked backto the internal free queue. In either case(OUTMBLK or PUTMBLK), once all thedataaretransmitted the message is returned to its originator. This codeis foll owed in the source by theFLUSHhandling describedabove. Then:

if (port->writeq.head){

port->currtxm = port->writeq.head;port->currtxp = port->currtxm->b_rptr;

}}

}

thevariablesaretheninitialised for thenext message at theheadof theoutput queue.

if (port->currtxm == NULL && todo == ’\0’){

WR(IER, IER_RxIE | IER_RLS);}

the‘transmitavailable’ interrupt is disabledonly whentherearenomoremessage,andno todo charactersto besent.

6.5.11 The RETMBL K Message

QueueHandler

rome_auto_reply(mptr);

Sincethe UART usesthe immediatedataareafor GETMBLK requests,thereis no processing neededtoreturn this buffer.

6.5.12 The main routine

Thereis no processingin the main routine that hasnot beencoveredabove in the message descriptions.However, it does introduceanother useful ROME routine. In many cases, main routines for processesperform identical logical operations, looping forever receivingmessages,decoding the operation codeandcalling a processing routine to handle that code. This is madeeasierby the rome_generic_handlerlibrary routine,which doesthedecoding automatically, givena tableof operationscodes:

46

6 TUTORIAL V: UART16550 6.6 TheDebugging Environment

static ROME_HANDLERS serial_routines[] ={

{ROME_M_CLOSE, serial_m_close},{ROME_M_FLUSH, serial_m_flush},{ROME_M_COMMAND, serial_m_cmd}

};

void serial_process(void)

{while (1 == 1){

rome_generic_handler(rome_await_message(0, 0), serial_routines,sizeof(serial_routines) / sizeof(ROME_HANDLERS));

}}

Theother messageacceptedby theserialmodule,COMMAND, will bedescribedin thenext tutorial.

6.6 The Debugging Envir onment

Debugging codein an embedded system is not easy, particularly when the operating system doesnotprotect individual process’s memoryandwhenthesystemis handling multimedia data.

6.6.1 Philosophy

ROME processes,which enjoy all thebenefits of shared memoryaccessfor fastdata transport, mustfacetheconsequenceswhenaprocesscancorrupt memorywhichcangoundetectedfor many machinecycles.The traditional approachto detecting suchproblemsis to run a process‘under the debugger’ andtraceinstructions and operations until the problem is found. The other traditional approachis to dump theprocess’statein a core file andattempt to reconstruct eventsleading up to thepoint of failure.

Neitherapproachworks well for ROME. Breakpointing codeonly works for non-time-critical applica-tions, wherepausing operations for tensof secondswhile a userpresseskeys on a keyboardis anaccept-ablepossibility . For a system handling real-time video, even 30msdelay would disrupt the dataflow totheextent thatany ‘normal’ operation could not beresumed. Likewise, in systemswith no external diskstorage,dumping core requiresa functioning network link, anda protocol stack, which maybe the verycomponentthatis beingcorrupted. ‘Fixing up’ thecodeto allow thenetwork dumpto completemayhidetheproblemevenmore.

ROME doesnot provide a traditional breakpointing debugger. The debugging environment is insteadconstructed from four components:

(a) amachine-level debuggerallowsaccessfrom theUART to thestateof themachine; its regis-ters,processesandmemory. Thisroutinetries,in general,to usecodethatis separatefrom themainprocessandruntimecode(for exampleit usesrome_kprintf andthepolled I/O routinesfor all its interactions with theuser). This routineis tied to variouserrorconditions detectedin theROME core(unhandledinterrupts,addressor dataexceptions).

(b) any routine may, at any time,placea ‘trace’ record in thesystem trace table. This records thelast256eventsin thesystem. It maybeused,for example,to recordthevaluesof variableson entry andexit from anISR or to monitorthecontext-switching flow through thesystem.

47

6.6 TheDebugging Environment 6 TUTORIAL V: UART16550

(c) thedebugger provided in (a) to catchsystemerrorsis available asa shared-library interfacethrough the rome_debug call. This effectively givesa form of breakpointing, under programcontrol.

(d) through the ROME imagesymbol table, the debugger can call user-supplied routines forspecificenvironments. For examplea module could supply a routinewhich formats its datastructuresfor humanconsumption. Thissimplifies thedebugger, whichknowsonly about thecorestructuresandallows a flexible anddistributedapproachto fault management.

The overall approachis to have the debugger catch errorsquickly, anddo aslittle aspossible to changethe stateoncethey have beendetected. The ‘shared library’ approachto entering the debuggerallowsamoduledeveloperto usethefull range of ‘C’ constructsin makingconsistency checks on data.

6.6.2 Enabling a debug envir onment

ThissectionintroducestheROMEdebuggerusingthedirectentry from theUART interrupthandler. UsingthePerf projectfrom tutorial III, enabletheSERIAL_UART16550_ENTER_DEBUGandROME_TRACE_INTERRUPTSoptionsin theproject definition, thenre-build.

Theeffect of ROME_TRACEINTERRUPTSon theserial interrupt handler is shownabove, it compilesinthecallsto rome_add_trace. Theothercodewasomittedfrom theearlier description, but is givenhere. Itis partof theinputr handling routine of theserial interrupt service routine:

#ifdef SERIAL_UART16550_ENTER_DEBUGif (uix == 0 && data == SERIAL_UART16550_ENTER_DEBUG){

rome_debug();}else

#endif

After rebuilding, do a make clean to remove the old objectfiles, thenmake, install andboot the system.It should produce almost the sameoutput as before (the ‘built’ string will be different of course) andcache-line effects maychangetheactual performancevalues.

ROME Initialising.Copyright 1997 NEC USA Inc.Built by leslie on pc-rome at Thu Apr 5 13:18:01 2001Starting the Scheduler.....perf: Queue-Handling 0.31

Pressing the‘!’ key on theterminalkeyboardwill now enter thesystemdebugger.

!Entering debugger with SP 0x00ffd7cc FP 0x00ffd7e4rome debug>

At this point the whole system hasstopped except for a routine polling the keyboardfor input. No in-terrupts arebeingprocessedandno messages arebeing handled. It is possible to examine andmodifymemoryin thesystem andto look at thestateof theprocesses.

48

6 TUTORIAL V: UART16550 6.6 TheDebugging Environment

6.6.3 Debug commands

Thesetof commandsavailablein thedebugger will dependon thearchitectureof thesystem (for examplethe MIPS version doesnot support call traceback, though the Intel versions do). Also information onregister andmemoryusewill vary from board to board. All systems provide the help commandwhichlists theavailablecommands.

For the symbolic information to be printed, the systemmustbe linked with the Createa symbol tableoption enabled in the‘Misc’ section of theproject description.

The lp commandlists theprocessesin thesystem:

rome debug> lpProcess console located at 0x00fffe70.Process serial located at 0x00fff4c0.Process up located at 0x00ffe310.Process resp located at 0x00ffd960.Process perf located at 0x00ffcfb0.Process timer located at 0x00ffc600.Process idle located at 0x00ffa450.Process pointer is currently pointing to resp.

dependingonexactly whentheinterrupt for the‘!’ key is handled,thecurrent processmaybeoneof perf,resp,uptime,console or uart.A particularprocessmaybeselectedwith thecpcommand(changeprocess)andthepinfo command prints information on thatprocess:

rome debug> cp idleSwitching process pointer to process idle.rome debug> pinfoProcess idle at 0x00ffa450Stack at 0x00ffa3d0EDI 0x00000000 ESI 0x00000000 EBP 0x00ffa3fc ESP 0x00ffa3f0EAX 0x002c6dfd EBX 0x00000000 ECX 0x00000000 EDX 0x002c6dfeEIP 0x00014558 CS 0x00000008 EFLAGS 0x00000213State: 1, Priority: 0No messages on queue.

The output showsthe processorregistersfor the process, its state (runnable)and its current schedulingpriority (0).

TheROME_TRACE_INTERRUPTSOptionturned on interrupt tracing for theUART. This writesa line inthetracetablefor everyentryto, andexit from, theUART ISR.Thetracecommanddisplaysthe(circular)tracetable:

rome debug> traceidle: Start interrupt 0x00000000 at 0x000000c2idle: End interrupt 0x00000000 at 0x00000000idle: Start interrupt 0x00000000 at 0x000000c2idle: End interrupt 0x00000000 at 0x00000000

this sample shows a smallpartof a full trace. Theoutput lists theprocessthatwasrunning at thetime ofthetraceentry, thetracetypeandthetwo parameterssupplied on thetracecall. Thedebuggerunderstands

49

6.6 TheDebugging Environment 6 TUTORIAL V: UART16550

certain pre-definedtracetypes, including thestartandendinterruptentries.Othertracetypesmaydisplayasa hexadecimalnumber. User-definedtracenumbers should startat 256(0x100). Usingtrace recordsitis possibleto observe theflow of control, andby recording thecurrent process,routinesin shared librariescanalsobedebugged.

In the above trace,notethat asthe ‘perf’ processis blocked waiting for the printf call to complete,theonly runnableprocessis idle, until thereply to themessageis sentby theUART ISR.

Thesymbols commanddisplaysthelocal symboltable:

rome debug> symbols__start 0x00010002_rome_start 0x0001406c__bssstart 0x00022020__bssend 0x00022310_cpu_interrupt_uh 0x0001122c_cpu_interrupt_error 0x00011080_icu_exception_handlers 0x00022240_rome_allow_reschedule 0x00021198_rome_this_ptr 0x00022300_rome_run_queue 0x00022304_cpu_scheduler 0x00010572_cpu_suspend 0x0001056c_rome_debug 0x0001058c

andthedefinedsymbols canbeused to examinememorylocations, for exampleto view code:

rome debug> di _rome_start0001406c: 55 U PUSH EBP0001406d: 89e5 .. MOV EBP,ESP0001406f: 83ec14 ... SUB ESP,1400014072: 57 W PUSH EDI00014073: 56 V PUSH ESI00014074: 53 S PUSH EBX00014075: c745fc00000000 .E..... MOV fc[EBP],000000000001407c: e8dbfbffff ..... CALL 13c5c00014081: e886490000 ..I.. CALL 18a0c

etc.

or variables:

rome debug> dm.w _rome_run_queue0x00022304: 0x00ffd960 0x00201000 0x00fffe70 0x1e3697cd

This showstheheadprocess(0xffd960, resp) linked into thequeue of runnableproceses.

Themessage commandwill formatanareaof memoryasa ROME message, treating thedataareaasthemblk form. This covers the majority of messagesin the system. For othermessage formats, moduleswhich definethemessagemayprovide routinesto formattheir own messages.

Symbolscanbeusedfrom thedebuggerto executesuch local routines.Usingdebugger-callable routinesallows thedebugger to bekeptsimple,sinceit does not needknowledgeof all theinternal datastructuresof eachof themodules,andallows eachmodule control over thedisplay andformatting of its data.Moreinformationabout thedebugger featurescanbefoundin theROME Porting Guide.

50

7 TUTORIAL VI: ETHERNETARP

7 Tutorial VI: Ethernet ARP

Thistutorial presentsmoduledesignedto sit within adataflow (unliketheUART modulewhichterminatesa flow). Themodule shows how theROME messagefieldsaremanipulatedfor zero-copy dataflowswithprotocol processing.

7.1 Functionality

The ETHER_ARP module implementsthe Internet Protocol layer 3 addressto ethernet MAC addresstranslation operation. It alsohandlesthe AddressResolution Protocol (ARP). The processoperatesbe-tweenoneor more‘downstream’ ethernetdriver processesandoneor more‘upstream’network protocolprocesses(usually only IP).Theprocessis responsiblefor supplying theethernetheaderaddressesfor out-bound packets, androuting themto thecorrect downstreamdevice. It examinestheheader for incomingpacketsandpassesthem upstreamto thecorrectnetwork layer (or handlestheminternally).

As with theUART module,ETHER_ARPreceivesrequestsfrom its upstreamclients. UnlikeUART, mostof theserequestsmustbe passed down to the appropriate driver. The replies to theserequestsalsopassthrough theETHER_ARPprocess.This is animportantaspectof thedataflow model.

7.2 FILEs

Both thehello processandtheecho processused ‘C’ runtimeI/O routineswhich take aFILE * parameter,either explicitly (aswith fgets) or implicitly (aswith printf ). The FILE is an important abstraction forsimplifying the downstream flow of data. Apart from the support for character-modeI/O (which is notrecommended,but is explainedat theendof thetutorial), a FILE just containsthelinking informationtopassmessagesdownstream:

typedef struct _file{

ROME_PROCESS *dest_pid; where this connectsvoid *opaque; returned by openint flags; eof etc.int fileno; file numberint buffer; for ungetc

}FILE;

Thedest_pid variable is theresultfrom calling rome_find_queueonthesuppliedfilename, andtheopaquevariable holds thedest_contextfield to beinsertedinto all messages sendfrom this file.

Even when the ‘C’ standard I/O library is not being used,the FILE is still a useful representation forinter-processflows. Theusualinterfaceto thesix dataflow messages is through a setof library routines,five of which: rome_fetmblk, rome_getmblk, rome_newmblk, rome_outmblk, androme_putmblkall take apointer to a FILE structureastheir first argument. Theother routine, rome_retmblk takes just a messagepointer.

7.3 URLs

Within a layeredprotocol stack, andparticularly for IP, thereis a fan-out of processestowardstheappli-cations. IP supportsmultiple transport protocols (notably TCPandUDP)andTCPsupportsmultiple user

51

7.4 Upstream,DownstreamandQueues 7 TUTORIAL VI: ETHERNETARP

connections. Theconnections aredistinguishedby small integers, representing procotol types andports.For exampleeachIP packet containsa protocol numberwhich distinguishesTCPtraffic from UDP (andICMP andother IP-based protocols). Similary, ethernet framesaredistinguishedby a 16bit type field intheheader. TheARP processitself doesnot need any knowledgeof thedetails of mostof theprotocols,but it doesneedto passthedatato theappropriateprocess.Thesameargumentholdsfor IP, UDPandTCP.Whena processopens a dataflow to theETHER_ARPprocessit mustindicatewhich protocol packets itwishesto process. This informationis passedto the fopen routine in theopenstring parameter, encodedin a form verysimilar to the‘Uniform ResourceLocator’ which is thedefactostandardfor representationof resourcesin theWorld-Wide Web.

ROME usesthe schemepart of the URL (the string before the first ‘:’) to identify a processwithin theimage.Theterminating ‘:’ is optional if no further informationis passed.For example:

fp = fopen("console", "rw");

contains a minimal ROME URL string consistingof just a processname.Thestandardschemesdefinedfor the URL do not have any place for information local to the originating system. ROME places thisinformationaftertheschemeterminator andbefore theinternetsyntaxpart.This is usedto hold local portor protocol numbers. TheROME URL syntax is:

process:localport//user:password@host:port/url-path

whereall fieldsareoptional exceptfor theprocess. Standard URLs useanencoding schemewith ‘%’ astheencoding escape character to allow, for example, ‘@’ in a password. Thestring form is only usedonthe interfaceto fopen for compatibility with standardC programmingconventions. Internally, andin theOPENmessage,a ROME URL is representedby thefollowing structure:

#define ROME_NAMELEN 32#define ROME_HOSTLEN 64

typedef struct{

char scheme[ROME_NAMELEN]; process nameuint port; local protocol idchar user[ROME_NAMELEN]; remote userchar password[ROME_NAMELEN]; remote passwordchar host[ROME_HOSTLEN]; remote hostnameuint ipaddr; remote host addressuint ip_port; remote portchar *urlpath; rest of path

}ROME_URL;

thevaluesin thesefieldsarenot URL-encoded.Therome_parse_url routineconvertsthestring form intothestructure.

7.4 Upstream,Downstreamand Queues

So far, ROME messages have beenshown passing betweenonly two processes. The sender had thesrc_context field available for its useandthe receiver hadthe dest_context field. Whenthereis a third

52

7 TUTORIAL VI: ETHERNETARP 7.5 Configuration Messages

processintervening, in theory it hasno placeto storecontext information.Any solution thatreservesonefield perpotential processpermessage doesnot scale.

If a processon a flow does not require local stateit canpassthe contexts betweenits adjacent pairs ofprocesses.Howevermany processes, includingETHER_ARP, needto accesscontext information,andtheonly option is to place it in themessage.

ROME adopts a model that hassimilar behaviour to the STREAMS modulestack, but with a differentinternal implementation. Thelink between threeprocessesis representedby a queue_tdatatype:

typedef struct{

ROME_PROCESS *src; uplink pointerROME_PROCESS *me; my pidFILE *dest; downlink pointerROME_MQUEUE reads; reads listptr q_ptr; local context

}queue_t;

this links a upstreamprocessto a downstreamfile, andcontains a pointer to thelocal context informationfor theflow.

As eachmessagepasses through theprocess,thesource context field is saved(downstream)andrestored(upstream)usinga local block:

typedef struct _context{

struct _context *link; free queue linkingqueue_t *local_context; this process’ informationptr src_context; saved source context

}CONTEXT;

A pointer to this datastructure is placed in the src_context field in the downstream direction. Theseoperationsare implemented by a pair of routines which perform approximately the samefunction asthe putq family of routinesfor STREAMS services. The rome_pass_downstreamroutine passesa mes-sageto the processrepresented by the dest file in a queue structure, saving the necessarycontext. Therome_pass_upstream routinerestorestheupstreamcontext andpassesthereply to thesrc process.

TheCONTEXT memoryis managed from a sharedpool acrossall processesanddoesnot require a mal-loc/freeoverheadonce thesystemis in a steadystate.

OnemajordifferencebetweenROMEandtraditionalSTREAMS is thatROMEdoesnotrequireaseparatescheduler to run STREAMS modules. The corescheduler runs the processesdepending on the stateoftheir queues. There is no concept of enabling or disabling queues, though the STREAMS emulationlibrary supportsthevarious routines;they just have no effect.

7.5 Configuration Messages

TheETHER_ARPmoduleis completely portable between all (current) ROME platforms. It hasno spe-cific knowledgeof thelower level interfacesor drivers.For themostpartit operatesonly ontheninebasicdataflow messages. It does,though,needsomemachine-specific informationin orderto operatecorrectly.

53

7.6 Finite StateMachines 7 TUTORIAL VI: ETHERNETARP

Specifically, it needs to know what arethe downstreaminterfaces,andwhat aretheir local interfacead-dresses. This is very similar to the ifconfig operation on UNIX systems. Ratherthandefinea new setof messagesto carry this information,ROME uses a single, simplemessage, calledCOMMAND. Thismessage carries a text string into theprocess,whereit canbeinterpretedasrequired.

The advantageof using a message insteadof a shared-library call is that the messagecanbe compiledin to the system even if the moduleis not present. Becauseprocessesarealsoidentified by strings, therome_find_queuecall will returnanerror if theprocessis absent, andthemessage will not besent. If thiswasa shared library, it would benecessaryto conditionally compile in theconfigurationcallsonly if themoduleis present.

TheCOMMANDmessageworkswell for setting information,however it doesnot return any information.In orderto extract informationfrom aprocess,anexplicit messagemustbesent.Thenetwork stack definesits own MessageSet,NetInfo, which is usedbetweenthenetwork modulesto passinformationaround. ForexampletheNETADDRESSmessageis usedto readthelocal MAC (layer-2) addressof a driver.

7.6 Finite StateMachines

In addition to passing ethernetpacketsup to IP, theETHER_ARPmoduleimplements ARP – in order tofind theethernetaddressesin thefirst place. If anIP-Address to MAC addressmapping is not known, theprocesssends out an ARP requestto the network andwaits for the reply. Sincepacketsmay get lost inthenetwork, theprocessmustalsore-transmitpacketson timeouts. Therepliesarecachedto reducethenumberof requests sentto thenetwork. This meansthata givencacheentrycanbein oneof a numberofpossible ‘states’ (suchaswaiting for a reply to anARP request).Transitionsbetweenstatesare(usually)triggeredby messagesarriving for theprocess,for examplea timeoutto causea re-transmission.

The natural representation of this mechanism is asa formal Finite StateMachine. In the sameway thatROME support message dispatching, through the rome_generic_handler routine, there is basic supportfor FSMs. However, this will not becoveredexplicity in this tutorial, to keepthe focus on themessage-passing interface.

7.7 Events

Someprocessescangenerateasynchronous ‘events’. A typical event of this form is the ejection or in-sertion of removablemedia.As well astheobviousfloppy disks, this could alsoincludePCMCIA-basedinterfacecards,including ethernetcards. In asimilar apprach to configurationcommands,sucheventsarepassed around the system using messages. All eventsarerepresented by a single message with opcodeEVENT. Thefirst parameterof eacheventmessage is a ‘type code’which identifies theexacteventbeingsupplied.

7.8 SourceCode

As with the UART tutorial, the codewill be described message by message. Therearetwo differencesfrom UART: themoduledoesnotcontainaninterrupthandler;andthemodulemustprocessbothmessages(from upstream)andreplies(from downstream).

54

7 TUTORIAL VI: ETHERNETARP 7.8 SourceCode

7.8.1 Data Structur es

Eachflow through the ETHER_ARP processis representedby an queue type (seeabove); eachconfig-uredinterfacebelow ETHER_ARPis representedby a ARP_DOWNINT type andeachARP entry by aARPENTRY type.

typedef struct{

int state; state of entrymblk_t *mp; mblk of pending ARPsint nr_arp; number of ARPs to tryint epoch; set timeint ifno; interface to send it onuint ip; the IP addressMACADDR mac; MAC address

}ARPENTRY;

typedef struct{

FILE *arp_fp; file descriptor for ARPschar ifname[16]; interface nameint ifno; interface numberMACADDR mac; our MAC addressuint subnet_address;uint ip; our IP addressuint ipmask; and maskARPENTRY broadcast; for broadcast pktsARPENTRY *local; The ARP tableint localsize; size of the table

}ARP_DOWNINT;

thearrayof interfacesis held in local static storage:

static ARP_DOWNINT *di[EARP_MAX_INTERFACES];

Thedi entries areassignedastheconfiguration commandmessagesareprocessed.

7.8.2 The CommandMessage

TheETHER_ARPprocessmaybeconnectedto multiple ethernet interfacecardsin a system.Thesemaybedifferenttypesof cards, with different interfacedrivers, or multiple instancesof thesamecardsharingthesamedriver. Commandmessagesareusedto configure the interfacesfor a particular machine. Theyareall handledby themainprocess.

interface: interfaceinterface-index device-url interface-number

if (sscanf(inc, "interface %d %s %d", &i, ifname, &i0) == 3){

ROME_MESSAGE *m;int j;ROME_URL dest;

55

7.8 SourceCode 7 TUTORIAL VI: ETHERNETARP

The interfacemessagelinks an interfaceindex number used by IP to a particular ethernet card in thesystem. Thecardis identified by a device URL anda ‘selector’ (theinterfacenumber) within thatdevice.For exampleamachinewith two cardssharing thesamedrivermightbenumbered‘0’ and‘1’ on thesameURL.

di[i] = rome_alloc(sizeof(ARP_DOWNINT), ’e’, 1);strcpy(di[i]->ifname, ifname);di[i]->ifno = i0;strcpy(dest.scheme, ifname);dest.port = i0;dest.ip_port = ETHER_TYPE_ARP;

In theURL passed to anethernet interface,thedevice selector is in theport field, andthe ip_port field isusedto contain theprotocolselector, in this caseETHER_TYPE_ARP (0x0806)to select only theethernetARPpackets.

if ((di[i]->arp_fp = rome_open_url(&dest, "rw")) == (FILE *)NULL){

rome_fatal("Couldn’t open ethernet interface\n");}

if (i+1 > dc) dc = i+1;

By usingthe FILE interfaceto thedevice, theETHER_ARPprocessallows theethernetdriver to setitsown context variables for theflow andreturn themhere. rome_open_url is the ‘internal’ routine used toopena file usinga pre-parsed (or explicitly constructed) URL. The removesthe needto convert a URLbackto a stringform in orderto usefopen(which would then have to parse it again).

earp_reinit(i);for (j = 0; 10 > j; j++){

m = (ROME_MESSAGE *)rome_alloc(sizeof(ROME_MESSAGE), 0, 1);m->src_context = (ptr)i;rome_getmblk(di[i]->arp_fp, m);

}}

Simply opening the interfaceFILE doesnot causepackets to arrive at the process. The ETHER_ARPprocessmustexplicitly request packetson the file. In this case, up to ten incoming ARP messageswillbequeuedwithin thesystem. Notethat it is essential that the rome_getmblkroutinedoesnot wait for thereply, otherwise the processwould block after the first requestwasissued. Also, since thesemessageswill remainin thesystemafterthis routine returns,they mustbeallocatedoff theheap, not thelocalstack.In order to determinefrom which interfacea reply hascome,theprocessputs theinterfaceindex numberinto thesourcecontext field of themessage.

Thecodecalledabove to (re-)initiali setheinterfaceis:

static void earp_reinit(int i){

ROME_MESSAGE m;ROME_T_NETADDRESS *na = RCAST(ROME_T_NETADDRESS, m);

56

7 TUTORIAL VI: ETHERNETARP 7.8 SourceCode

m.opcode = ROME_M_NETADDRESS;m.dest_context = di[i]->arp_fp->opaque;rome_sendwait(&m, di[i]->arp_fp->dest_pid);

Thereis no shared-library interfacefor the NETADDRESSmessage(since it is only used in oneplace),so the message is constructed explicitly. In order to identify the interfaceto the lower layer, the correctdestination context mustbeplaced in themessage. This is copiedfrom theFILE structure,whereit wasplaced by rome_open_url. rome_sendwait is a library routine that completesthe fields of the message,sends it to thedestination, andwaits for the reply. In this case, therefore, the messagecanbeput on thelocal stack.

if (m.m_errno){

rome_kprintf("EARP: no MAC address for interface %d\n", i);rome_fatal("interface error");

}memcpy(di[i]->mac, na->mac, 6);

In theory, no ethernet interface can run without knowing its own MAC address. If, for somereason,the destination processreturnsan error, thenthe system haltsthrough the rome_fatal routine. This is anexampleof thedebugging philosophy described in theprevious tutorial. Sincethis probably representsatyping error in theconfigurationcommand,rather thantry andhidetheerror, thesystem stops asquicklyaspossibleto allow a userto examinethestateandcorrecttheproblem.

if (na->events_generated){

int j;ROME_MESSAGE *tm;

for (j = 0; 2 > j; j++){

tm = (ROME_MESSAGE *)rome_alloc(sizeof(ROME_MESSAGE), 0, 1);tm->src_context = (ptr)i;rome_get_event(di[i]->arp_fp, tm, rome_this_ptr->prio);

}}

}

If the interfacereports that it supportsevents (seeabove), thentheETHER_ARPprocesssetsup a queueof eventrequeststo thatprocess,in a similar mannerasthedatarequests.

config: configinterface-numberipaddressnetmask

if (sscanf(inc, "config %d %I %I", &i, &i0, &m0) == 3){

The config commandcompletesthe initi alisation of an interface,by supplying the local IP addressandnetmask for that interface.

57

7.8 SourceCode 7 TUTORIAL VI: ETHERNETARP

if (i < 0 || i > EARP_MAX_INTERFACES || di[i] == NULL){

rome_fatal("EARP: Bad interface in config");}

Theinterfacemustpreviously have beensetup by an interfacecommand.

di[i]->ip = i0;di[i]->ipmask = m0;di[i]->subnet_address = i0 & m0;di[i]->broadcast.state = ARP_ENTRY_OK;di[i]->broadcast.ifno = i;memset(di[i]->broadcast.mac, 0xff, 6);di[i]->localsize = ntohl(~m0) + 1;di[i]->local = rome_alloc(di[i]->localsize*sizeof(ARPENTRY), ’e’, 1);

}

The netmask is usedto determine the size of the routing subnet attached to that interface,and so tocalculatethepossible sizeof theARPtable.

7.8.3 The EVENT Message

QueueHandler

rome_pass_downstream(q, mptr);return ROME_HANDLED;

SincetheETHER_ARPprocessdoesnot itself generateevents, any eventrequestsarepasseddownstreamto thedriver.

7.8.4 The EVENT Reply

QueueHandler

if ((uint)mptr->src_context < EARP_MAX_INTERFACES){

return ROME_NOT_HANDLED;}rome_pass_upstream(mptr);return ROME_HANDLED;

The ETHER_ARP processmakesuseof the fact that EVENTmessagespassed downstreamwill have apointer valuein thesrc_context field, which cannot bea small integer. Therefore if thevalueis oneof theinterfaceindex numbers, it musthave originatedwithin theETHER_ARPprocessitself, in which caseitshould behandledin themainprocess.

Main Process

58

7 TUTORIAL VI: ETHERNETARP 7.8 SourceCode

ROME_T_EVENT *event = RCASTP(ROME_T_EVENT, mptr);int ifno = (int)mptr->src_context;

#ifdef ROME_E_PCMCIA_CARD_INSERTif (event->event_type == ROME_E_PCMCIA_CARD_INSERT){

earp_reinit(ifno);}#endifrome_get_event(di[ifno]->arp_fp, mptr, rome_this_ptr->prio);

This code re-initialises theMAC addressfor a newly-i nserted PCMCIA ethernetcard.

7.8.5 The OPEN Message

Main Process

ROME_T_OPEN *o_msg = RCASTP(ROME_T_OPEN, mptr);queue_t *q = (queue_t *)rome_alloc(sizeof(queue_t), 0, 1);ROME_URL dest;int ix = o_msg->openurl->port;

if (ix >= dc || di[ix] == NULL){

rome_free(q);mptr->m_errno = ENXIO;

}

TheURL containsthe interfaceindex in theport field andtheprotocol selector in the ip_port field. Theindex mustcorrespondto a previously-configureddownlink.

else{

strcpy(dest.scheme, di[ix]->ifname);dest.port = di[ix]->ifno;dest.ip_port = o_msg->openurl->ip_port;q->src = mptr->src;q->me = rome_this_ptr;q->dest = rome_open_url(&dest, "rw");mptr->dest_context = q;

}rome_reply(mptr);

If it does, theindex is usedto constructanew URL based onthedownwardinterfacename.This is openedasa file which is usedto initi alisethethree-way queuestructure,which is returnedasthelocal context tothecaller.

7.8.6 The CLOSE Message

Main Process

59

7.8 SourceCode 7 TUTORIAL VI: ETHERNETARP

queue_t *q = (queue_t *)mptr->dest_context;

fclose(q->dest);rome_free(q);rome_reply(mptr);

Theprocessing for theCLOSEmessageis simplerthanfor theUART code,astheprocessdoesnot keepa list of readrequests. It is assumedthat the higher layer process does not try to closea file that hasoutstanding write requests. If so, the codewould needto check that no messages for this queue arecurrently heldawaiting ARPreplies. As it stands, thecodesimply closesthedownstreamfile (which maycauseit to return message upwards) andfreesthequeue structure.

7.8.7 The FLUSH Message

QueueHandler

rome_pass_downstream(q, mptr);return ROME_HANDLED;

It is known that the IP codedoesnot generateFLUSH messages,andasthat is theonly processthatcancausedatato bebufferedinside theETHER_ARPprocess,themessagecanbesimply passed down to thelower layer.

7.8.8 The FLUSH Reply

QueueHandler

rome_pass_upstream(mptr);return ROME_HANDLED;

Sinceall FLUSHmessageswerepasseddownstream,all repliesmustbepassed upstream.TheETHER_ARPcodedoes not generate FLUSH requestsitself, sothere is no needto checkfor repliesto local messages.

7.8.9 The FETMB LK and GETM BLK Messages

QueueHandler

rome_pass_downstream(q, mptr);return ROME_HANDLED;

Requests for filled buffersarepassed downstream,sincetheethernetdriver hassufficient information(theethertypeprotocol selector) to assign incoming framesto individual flows.

60

7 TUTORIAL VI: ETHERNETARP 7.8 SourceCode

7.8.10 The FETMBL K and GETMBL K Replies

QueueHandler

if ((uint)mptr->src_context < EARP_MAX_INTERFACES){

return ROME_NOT_HANDLED;}

Repliesto messagesthat weresent from within the ETHER_ARPprocessitself arepassedto the mainprocess(below).

mptr->b_rptr += sizeof(ETHER_HEADER);rome_pass_upstream(mptr);return ROME_HANDLED;

Theethernetdriver returnsthe‘readpointer’ for theincoming data at thestart of theethernetframe.TheETHER_ARP processmodified this pointer to point to the start of the layer-3 protocol header beforepassing themessageupwards. In thisway theIP codeis unawareof thesizeof any headeron themessage(sotheIP codeis independent of thelayer-2 addressingmodes).

Main Process

ETHERARP *arph = (ETHERARP *)mptr->b_rptr;int ifno = (int)mptr->src_context;

if (ntohs(arph->ar_op) == ARPOP_REPLY){

ARPENTRY *entry = earp_arp_get(*(uint *)arph->arp_spa);

if (entry){

(*earp_p_machine[entry->state])(ARPEVENT_RESOLVED, entry, mptr);}

}

An incoming message from the net could be a reply to a request sent out from this machine. If so, theFinite StateMachinefor thatrequestis run with theARPEVENT_RESOLVEDevent.

else if (ntohs(arph->ar_op) == ARPOP_REQUEST){

uint forme = *(uint *)arph->arp_tpa;

if (forme == di[ifno]->ip){

or, it couldbea requestfrom another machine for oneof our interfaceaddresses. Thecodeonly replies iftherequestcamein on thesameinterfaceasits configuredIP address,to preventMAC addressescrossingsubnet or VLA N boundaries.

61

7.8 SourceCode 7 TUTORIAL VI: ETHERNETARP

ROME_MESSAGE reply_msg;ETHERARP *arp_reply;

reply_msg.src_context = (ptr)ifno;rome_newmblk(di[ifno]->arp_fp, &reply_msg, sizeof(ETHERARP));rome_await_message(&reply_msg, 0);

In orderto sendaresponseover thenetwork with theMAC address,theprocessmustrequestabuffer fromtheethernetdriver. It sendsaNEWMBLK messagedownstreamusing its FILE pointer, requesting abufferlarge enough to hold anETHERARPpacket. Thesourcecontext is markedwith theinterfacenumber(sothereply is passed backby theQueueHandler). Theprocessthenwaitsfor thereply.

arp_reply = (ETHERARP *)reply_msg.b_rptr;arp_reply->ar_hrd = arph->ar_hrd;arp_reply->ar_pro = arph->ar_pro;arp_reply->ar_hln = arph->ar_hln;arp_reply->ar_pln = arph->ar_pln;arp_reply->ar_op = htons(ARPOP_REPLY);memcpy(arp_reply->arp_sha, di[ifno]->mac, 6);memcpy(arp_reply->arp_tha, arph->arp_sha, 6);*(uint *)arp_reply->arp_spa = *(uint *)arph->arp_tpa;*(uint *)arp_reply->arp_tpa = *(uint *)arph->arp_spa;

the reply buffer, starting at b_rptr (which is the sameas b_wptr for an empty buffer), is cast to theappropriate typeandthefieldsarefilled in from theoriginal requestandthelocal information.

reply_msg.b_wptr += sizeof(ETHERARP);rome_putmblk(di[ifno]->arp_fp, &reply_msg);rome_await_message(&reply_msg, 0);

the b_wptr field is incrementedto point to the endof the valid data,and the message is sentusing therome_putmblk routine. In order to simplify the processingof ARP responses(andto allow the messageto beput on thelocal stack) theprocesswaitsfor thereply. Sincethis usesPUTMBLK, theethernet layerwill automatically freethebuffer after it hasbeentransmitted.

}}rome_retmblk(mptr);rome_getmblk(di[ifno]->arp_fp, mptr);

However, thebuffer which containstheoriginal requestmustbeexplicitly returnedto thedriver, since itwasreceived with a GETMBLK message. Oncethe buffer hasbeenreleased,the samemessagecanbere-usedto request another frame.

7.8.11 The NEWMBLK Message

QueueHandler

mptr->b_wptr += sizeof(ETHER_HEADER);rome_pass_downstream(q, mptr);return ROME_HANDLED;

Requests for emptybuffers arepassedto thedriver. Therequestedlength is increasedto allow roomforthe(fixed-size) ethernetheader to beaddedlater.

62

7 TUTORIAL VI: ETHERNETARP 7.8 SourceCode

7.8.12 The NEWMBLK Reply

QueueHandler

if ((uint)mptr->src_context < EARP_MAX_INTERFACES){

return ROME_NOT_HANDLED;}mptr->b_wptr = mptr->b_rptr += sizeof(ETHER_HEADER);rome_pass_upstream(mptr);return ROME_HANDLED;

whena reply to a NEWMBLK messageis received,therearetwo possibiliti es.If theblock wasrequestedby ETHER_ARPitself (for ARP responses), the message is handled by the main process. The localrequestsaredetected by having the src_context field contain a small integer rather than a pointer to aqueue structure. In this case, theROME_NOT_HANDLED indicationis returnedto thecore, to cause themessage to bescheduledfor themainprocess.

If themessagewasarequestby anetwork processaboveETHER_ARP, thepointersaremovedwithin thebuffer to leave roomfor theethernet header, thenthereply is thenpassed upstream.

7.8.13 The OUTMBLK and PUTMBL K Messages

QueueHandler

queue_t *q = (queue_t *)mptr->dest_context;

if (mptr->b_rptr != mptr->b_wptr){

thehandler only looksat messages thathave any actual content. This automaticallybypassesthePUTM-BLK messagesusedto return unusedbuffersto thedriver.

mptr->b_rptr -= sizeof(ETHER_HEADER);

if (M_IPDATA == mptr->b_type){

ETHER_HEADER *eh = (ETHER_HEADER *)mptr->b_rptr;ARPENTRY *entry = earp_arp_get(mptr->b_immed);

the IP layer is responsible for IP routing,anddetermining thecorrect interfacefor a message. This givestwo possible casesfor a message: either the addressin the IP header is to be usedas the destinationaddress(i.e. inside thesubnet); or thepacket is destinedfor a gateway on thesubnet, andthe IP addressin the header should not be usedto routethe packet. IP solvesthis issueby using the b_immedfield inthe message to contain the IP (version 4!) addressof the next-hop. To signal this to the ETHER_ARPlayer, themessagetype is setto M_IPDATA. TheETHER_ARPqueuehandler uses this value to find thedestination MAC address.

63

7.8 SourceCode 7 TUTORIAL VI: ETHERNETARP

if (entry == NULL){

rome_kprintf("ARP: %I not in our subnet?\n", mptr->b_immed);rome_fatal("ARP Routing Error");

}

becauseof this two-level scheme,theaddress should alwayslie within thesubnet, it is anerror in the IPlayerif it doesnot.

if (ARPSTATE_OK != entry->state){

return ROME_NOT_HANDLED;}memcpy(eh->ether_dhost, entry->mac, 6);memcpy(eh->ether_shost, di[entry->ifno]->mac, 6);

if theMAC addressof thatIP addressis notknown,themessageis passedon themainprocess. Otherwisetheaddressesaresetin theethernetheader (whichis madeaccessible by moving theb_rptr field backwardsin thebuffer).

}}rome_pass_downstream(q, mptr);return ROME_HANDLED;

oncetheaddressesareset,themessage is passedon to thedriver.

Main Process

ARPENTRY *entry = earp_arp_get(mptr->b_immed);

(*earp_p_machine[entry->state])(ARPEVENT_LOOKUP, entry, mptr);

Themainprocessis scheduled only whentheMAC addressis of thedestinationis not known. Thecodeinjects a LOOKUP event into thestate machinefor thatIP address.

7.8.14 The OUTMBL K and PUTMBLK Replies

QueueHandler

if ((uint)mptr->src_context < EARP_MAX_INTERFACES){

return ROME_NOT_HANDLED;}mptr->b_wptr = mptr->b_rptr += sizeof(ETHER_HEADER);rome_pass_upstream(mptr);return ROME_HANDLED;

Heretoo, repliesto OUTMBLK andPUTMBLK messagescould have comefrom locally-generatedtrafficidentified by thesrc_context field, or they could have comefrom anupstreamprocess.

Main Process

Thereis no explicit handler for thesereplies in the main process. The messages cause the process toresumeexecution from therome_await_message call in theF/GETMBLK reply codeabove.

64

7 TUTORIAL VI: ETHERNETARP 7.8 SourceCode

7.8.15 The RETMBL K Message

QueueHandler

rome_pass_downstream(q, mptr);return ROME_HANDLED;

returnedbuffers arepasseddown to thedriver which originated them.

7.8.16 The RETMBL K Reply

QueueHandler

if ((uint)mptr->src_context < EARP_MAX_INTERFACES){

return ROME_NOT_HANDLED;}mptr->b_wptr = mptr->b_rptr += sizeof(ETHER_HEADER);rome_pass_upstream(mptr);return ROME_HANDLED;

Heretoo, replies to RETMBLK messagescould havecomefrom locally-generatedtraffic identifiedby thesrc_context field, or they couldhave comefrom anupstreamprocess.

Main Process

Thereis no explicit handler for these replies in the main process. The messagescause the process toresumeexecution from therome_retmblkcall in theF/GETMBLK reply code above.

7.8.17 The TIMEOUT Reply

TheARP Finite StateMachine (which hasnot beendescribedhere) usestimeout calls to control requestretransmissions.Thetimeoutrepliesarehandled by a line in thehandlerslist:

{ROME_R_TIMEOUT, timer_tmhandler}

which automatically calls the timer_tmhandler routine,which should befamiliar from thepreviousmate-rial.

7.8.18 The main process

The main processis constructedaround a call to rome_generic_handler in the sameway as the serialmodule(but with a rather longer list of messages).

65

8 TUTORIAL VII: NETWORK ECHO

8 Tutorial VII: Network Echo

This tutorial . unlik e the previous ones,requiresspecificPC hardware and may not work in yourenvir onment.

Thisfinal tutorial bringsthenetworking componentstogetherby re-working the‘echo’ module to processdatafrom thenetwork. Thefocus for this tutorial is usingthemodulepagesto constructasystem from thedocumentedcomponents. Theaim is to attachtheIP stack under theconsole process andusethe‘telnet’protocol to connectto themachineover theethernet.

Theprocessstructure is shown in figure4. How thesemodules areknownto berequired is described inthenext section.

Core Interface API

core/"idle"cpu-plugin

icu-plugin

C Runtime Library

debugger

logical message path

actual message path

timerDC21041ethernet

I440BXPCIbus

TCP

telnet

Console

netecho

UART IRQ

ROME base system

ETHER_ARP

IP

Figure4: ModuleStructurefor Tutorial VII

8.1 Choosingthe Modules

Building a complete system for anembeddedapplication generally requiresa detailedknowledgeof theplatform on which theapplication is going to run. Theprevioustutorials werecarefully chosen to run on“almost all” PC-basedsystems. However, assoon asapplications require specific hardware, the picturechanges.For thePCdevelopmentof theROME system, thehardwareplatform wasaNECDirection SPB-450PC.This usesanIntel ‘Seattle’ motherboardwith a 450MHz Pentium-II processor. Themotherboardprovidesfour PCIbus slotsunder the control of the Intel I440BX chipset. The PC doesnot comewithonboardethernet,andaKingston KNE110TXPCIbuscardwasusedto make a local ethernetconnection.

All theROMEmodulesdescribehow they areto beusedandwhatthey canbeused for. Theether_dc21041_liteonmoduledescribes itself as a driver for the LiteOn clone of the DC21041‘Tulip’ chip, as used in theKingston KNE110TXethernetinterfacecard. As thecardis aPCIcard,aPCIbuscontroller modulemustalsobeincluded,in this casethepci_i440bxmodule.

At theother end, the telnetmodule hasa requirementon theTCPmodule, which itself usesthe IP mod-ule. Finally, the ETHER_ARPmoduleis the link between IP andan ethernet driver. The ethernet andETHER_ARPmodules bothspecify theNetInfomessagesetasa requirement(asdescribedabove).

66

8 TUTORIAL VII: NETWORK ECHO 8.2 Initialising IP

This gives enough information to generate the project description in RTB. Using the echo project oftutorial II, addthefollowing into theproject:

MessageSet: NetInfo, Timer

Modules: Driver.PCI.pci_i440bx, Driver.Timer.timer_pc, Driver.Ethernet.ether_dc21041_liteon, Net-work.Protocol.ETHER_ARP, Network.Protocol.IP, Network.Protocol.TCP, telnet

As before, the echo processis tied in to the console through the standard C I/O files, by marking theprocessto undergo C runtime initialisation.

8.2 Initialising IP

Themainapplication file, echo.c, needsa smallmodification to completethenetwork initi alisation. Theprevious chapter described the COMMAND message usedto link the IP protocol stackto a particularinterfaceand interfaceaddress. This message must be sent to IP to complete the link. The interfaceaddressis the“earp” process;theIP addressis statically linkedinto theprocess in this (simple) example.You should choose anIP addressallocatedto your domain. If it is not in thesamesubnet asthemachinewhich will startthetelnet connection, you will alsoneedto give a gateway address.

8.3 SourceCode

Sinceyoucreatedtheechomodule, it is already writable within themodule. Rememberto edit theversionin the hello/Modules/Echo direcroty, not the Sourcesdirectory. For example, to set the machine’s IPaddressto 138.15.103.243, addthelines:

rome_send_command("earp", "interface 0 ether 0");rome_send_command("ip",

"config 0 138.15.103.243 255.255.255.0 earp:0 2048");

to themainechoprocess.Theformatof thefirst string is explainedin theETHER_ARPtutorial, andtheformat of the second string is given in the IP moduledocumentation. The rome_send_commandroutineformatsandsendsa COMMAND messageto thespecifiedprocess.

8.4 Build, Load and Run

Theprogramis built andloadedin theusualway. Fromanothermachineonthesamenetwork (for examplethedevelopmenthost)a TCPconnection canbeopened over theethernet:

telnet 138.15.103.243Trying 138.15.103.243...Connected to 138.15.103.246.Escape character is ’^]’.logon: romeecho fooecho: fooecho this is a test of network echo

67

9 LEGACY APPLICATIONSAND THE C RUNTIME

>echo: this is a test of network echo^]telnet> closeConnection closed.

Thenetwork link over telnet promptsfor a login name,to prevent accidental connections to developmentsystems.Only thestring‘rome’ is accepted.Theconsole automatically redirectsall standardoutput to thenetwork link. Theconnectionis closed by enteringcontrol-] andthe‘close’ commandto the(local) telnetprocess.

9 LegacyApplications and the C Runtime

In anideal world theprevious tutorial would have completedall the functionality for theROME system.Thereremain,though, a few caseswhich do not fit into this model. Therearealso someapplicationswhich might seemnot to fit, but do indeed work well with ROME dataflows.

9.1 Traditional Bufferin g

Themain problem comeswith legacy applications which usethe C runtime library routine to readdata.An exampleis the echo programabove. It usesfgets to readdatainto a buffer it supplies. As expected,theC library passesthis to thedestinationprocessasa FETMBLK message:

char *fgets(char *s, int n, FILE *stream){

ROME_MESSAGE message;char *rv = s;

if (stream->buffer != EOF){

*s++ = stream->buffer;stream->buffer = EOF;n--;

}mblk_setup(&message, s, n);rome_fetmblk(stream, &message);rome_await_message(&message, 0);return ((message.m_errno == EOF) ? NULL : rv);

}

Thecodefor thestream->buffer variable handlesasinglecharacterpushedbackontothestreamby ungetc.The remainder of the codeformats a FETMBLK message, sendsit downstreamandwaits for the reply.Theproblemcomeswhenthis meetsa modulewhich operatesin theGETMBLK mode,for exampleIP.Ultimately, there is no choice but to copy thedata.

In echo, these two dataflows meet in the console module, which is wherethe datacopy occurs. Theconsole module alsohasto strip off theprefix for line-by-line input.

9.2 Buffered Output

Theoutput direction is lessof a problem. Therearetwo cases, printf, which usesan internal buffer, andfputswhich is givena string.

68

9 LEGACY APPLICATIONSAND THE C RUNTIME 9.2 BufferedOutput

9.2.1 printf

Theprintf routine works very well with ROME dataflows, since it canuseNEWMBLK andPUTMBLK.Theroutine which integratesthedifferent interfacesis vfprintf. This is calledby theotherroutines,suchasprintf andfprintf, with a uniform interface:

int vfprintf(FILE *stream, const char *format, va_list args){

ROME_MESSAGE msg;int size;

rome_newmblk(stream, &msg, MAX_PRINT_BUFF);rome_await_message(&msg, 0);

if ((size = gvsprintf(msg.b_wptr,(char *)format, args, MAX_PRINT_BUFF)) >= 0)

{msg.b_wptr += size;rome_putmblk(stream, &msg);rome_await_message(&msg, 0);

}return size;

}

theroutineblockstwice in rome_await_message until thereply is received,once for theNEWMBLK andoncefor thePUTMBLK. This style of programmingshowstheadvantageof handling theNEWMBLK intheQueueHandler, which savesa pair of context switcheson every call to printf.

9.2.2 fputs

The fputs routine generatesa OUTMBLK message:

int fputs(const char *s, FILE *stream){

ROME_MESSAGE message;int len = strlen(s);

if (NEQBIT(stream->flags, FILE_WRITE_FLAG)){

errno = EACCES;return EOF;

}mblk_setup(&message, (uchar *)s, len+1);message.b_wptr += len;rome_outmblk(stream, &message);rome_await_message(&message, 0);return 0;

}

The routine constructs an mblk with just enough spaceto contain the supplied string (allowing for theterminating NUL). This posesa smallproblemin theconsole processwheretheoriginating processnameneeds to beadded. With printf, spacehasbeenreservedat thestartof thebuffer (in exactly thesameway

69

9.2 BufferedOutput 9 LEGACY APPLICATIONSAND THE C RUNTIME

asthe IP modulereserved its header space). Underthese circumstances(andothers, for examplewhenmultiple linesarepassed in thesamebuffer), theconsole codewill copy thedatato another buffer. Thismerelyemphasisesthatthezero-copy featuresof ROME arenot automaticfor all programmingstyles.

This completesthetutorial material.

70