Chapter 5. Tools for Kernel Programmers

Chapter 5. Tools for Kernel Programmer sThekernelprogramingkitbox startswith theC compiler, but writing andcompilingcodeis only partofthestory. If youwantto codesomethingrelatively simple,andneedto getup andrunningquickly, thereareprototypingRAD tools,andto codeeffectively, youwill wantdebugging,traceandprofiling toolstoensureyourwork doeswhatyou expect.Finally, thereis no magicbullet to comprehendingwhereyourmodificationsshouldfit into Linux architecture,but therearesomeverygoodreverseengineeringtoolsto helpyou “call beforeyoudig”.

Note: This overview of programming tools, like most discussions on Linux, charts a moving target.New tools and updates are added almost daily; if your needs go beyond what is outlined here, do afresh hunt through one of the Linux kernel support sites.

Becauseit is essentiallyanembeddedapplication,workingwith thekernelmeansabandoningmanycomforttoolsof theprogrammer’stoolkit. You cannotusestandardlibc functionsor loggingtools;youhaveonly whatis providedby thecompilerandwhatis includedin thekernel.Fortunately, thereareotherwaysto getinsidethebox:Throughcarefulanalysisof thesourcefilesand,with minor restrictions,theuseof debuggersandprofilers,youcangetsomeinsideunderstandingof whatyourcodeis doing.

GCC, EGCS and GNU Assemb lerThefoundationof kernelprogrammingis theGNU CompilerCollection(GCC).Linux is definedby thetheinterpretationsof theC standardasthey appearin GCC;asa generalrule,Linux is portableto anyplatformwheretheFreeSoftwareFoundationhasportedtheGNU compiler. Thereareexceptions,butthis is theofficial relationshipbetweenLinux andGNU: All patchesandall bugreportsareverifiedthroughtheofficial, Linux-approvedversionof theGCC.

Linux programmingmaybegin with GCC,but not just any GCC:At thetimeof writing,Documentation/Changes lists version2.7.2.3astheonly compilerwherethekernelis expectedtocompile.Thesourcesarenot neccessarilyincompatiblewith otherversionsor evenwith otherCcompilers,but beforeyoucanreportsuspectedkernelor compilerbugs,youmustfirst verify your resultsusingthe2.7.2.3GNU compiler.

Note: Many core developers will test the kernel build with proprietary compilers (for example, theIRIX and Solaris C compilers) and with the Experimental GNU Compiler System (EGCS), and eachbuild will be tested in different configurations and platforms; unless you find problems specificallymentioned in the KernelNotes or kernel-list archives, the chances for success with another ANSIstandard C compiler are reasonably good.

1

Chapter5. Toolsfor KernelProgrammers

Lin ux 2.4 and egcsWhy not EGCS?After all, mostLinux distributionsreleasedafterthesummerof 1999bundletheEGCSversionof GNU C. This is frequentlydebatedin kernelcircles.Yes,EGCSis the“official” GNU C, butno, it is not thesameasGCC.Thekey word in “EGCS” is “Experimental”.

Someexplaination:After thereleaseof gcc2in 1997,theFreeSoftwareFoundationhaltedfurtherdevelopmentof thecompilerandtheEGCSgroupwasformedto carrytheprojectforward,this timeasatrueopensource initiative ratherthanasthecloisteredworkgroupmodelusedfor theearlierversions.InlateApril 1999,theFreeSoftwareFoundationformally appointedEGCSasofficial maintainersof thecompiler. As a result,EGCS,becauseit is “new technology”,is themostwidely distributedflavourof theGNU C compilerin moderndistributions.It is still "new technology"andthemergerof theold version2GCCcodeandthenewerEGCScodeis not yet complete.Therearesubtledifferences.

Strangethingscanhappenwith egcs,oftenverysubtleandhardto debug things.Themostsignificantdifferencebetweenthesecompilersis in optimization,especiallywith inline code.Prior to Linux 2.2,thedifferenceswerestark:Themoreaggressiveoptimizationson theIntel versioncausedwildly incorrectcompilations.Sincetheprimarygoalof the2.0/2.2kernelswasstabilityandEGCSpresentedtoomanyunknowns,thekernelgroupvotedto staywith 2.7.2.3until theseissuescouldberesolved.

This doesnotmeanyoucannotusethenewercompiler. Many problematicsectionshavebeenrewrittenandthemostgrossEGCScomplicationshavevanished.Still, absenceof evidenceis not thesameasevidenceof absence:As recentlyas2.3.18,optimizationproblemsagainpointedfingersatEGCSformiscompilationsof deeplynestedmacros.It is like thedentistwho tellspatientsthey neednot flossalltheir teeth,only thosethey wish to keep:YoucanuseEGCS,but if you requirestability, you reduceyourunknownswith theofficial, albeit lessclevercompiler.

Using GNU Assemb lerIf you feel theurgeto useassembler, getsomefriendsto talk youout of it. Most tasksarecompletelypracticalusingpureC, andmuchof theassemblercodein theLinux kernelhasmigratedto C to easeportability (or EGCSconflict) issues.If youabsolutelymusthaveclosecouplingwith yourhardwareandassembleris youronly option,then,justaswith theC compiler, theofficial assembleris theGNUAssembler(GAS) in thebinutils package.Onceagain,Documentation/Changes lists theofficial andminimumversionasbinutils-2.9.1.0.7; otherassemblersmaybepossible,andmany arementionedin theAssemblyHOWTO, but thesamecaveatapplies:Beforeyou canreportabug,youmustverify yourresultswith theofficial toolset.

Assemblercanbeincludedinline in theC sourcesor in stand-alonefiles;bothusethesameassembler, inonecaseinvokedby gcc, theotherby make rules.Theresultsandtherulesarethesame.Detailedinformationon mixing inline-assemblerwith C canbefoundin theGCCinfo pagesunderCExtensions:: � Extended Asm:: with detailsaboutIntel-platformconventionsunderInvokingGCC:: � Submodel Options:: � i386 Options::.

2


Inline Assemb ler and OptimizationBe very careful with compiler optimization flags when using inline assemblylanguage. GCC will not inline assembler functions unless optimization -O or -Onflags are added; without optimization, the link stage will hang looking forunresolved references to any assembly routines that may have been embedded inextern inline functions. You can mix -g and -O flags for debugging, but you mustinclude optimization for correct inline compilation.

If you know any otherassembler, youwill find few surprisesin GAS.GNU assemblerusesstandardAT&T syntax,a little oddto Intel users,but completelyfamiliar to thoseaccustomedto them68kassembler.

Assemb ler Caveats

TheAssembler-HOWTO offersthefollowing checkliston usingGNU assembler:

• Registernamesarealwaysprefixedwith thepercent(%) sign,asin %eax; thismakesit possibleto mixexternalC symbolswithout introducingunderscorenotation.

• Theorderof operandsis reversedfrom theIntel convention;gasexpectsoperandpairsas(source,destination)

• Operanddatatypescanbedisambiguatedby appendingasingle-lettertypeidentifierto thenameof theoperator, for example,

movw %ax,%dx

would movea 16-bit (word)value;thedefaultbehaviour will deducetheword-sizefrom theoperandsoruse32-bit (long)values.

• Immediateoperandsareidentifiedby a $ prefix

• Wherethereis no prefix,variableidentifiersaretakenasmemoryreferences,for example$var is theaddressof var whereasjustvar denotesthevaluestoredat thatlocation.

• Parenthesisarounda variablenamedenoteindirectionor indexing, for example17(%ebp) is anoffsetof 17 bytesfrom theaddressstoredin thebasepointer.

Intel2GAS Assemb ler Translator

In someinstances,assemblercodefrom otherplatformsandfrom otherassemblerscanbemachinetranslatedto GAS formatusingMikko Tiihonen’s Intel2GAS(ttp://hermes.terminal.at/intel2gas/)

3


translator. Originally designedto translateNASM files to theGNU Assembler, this projecthasevolvedinto a generalpurposesystemto translateseveralix86 assemblersinto GAS andcanbescriptedtotranslatebetweenalmostany two assemblersyntaxconventions.

Navigational AidsYou mayunderstandthecompilerandhavea goodnotionwhatyourcodemustaccomplish.Next, youneedto know whereandhow it will fit into thekernelarchitecture.Simply addinga new kerneldevicemodulemayhaveanobviousbest-location,but if yourprojectspansmoduleor subsystemboundaries,you needthebig picture.Evenif youareonly extendingexistingcodeor addinga new minorservice,without acurrentandaccuratemap,you runa risk of re-inventingserviceswhich mayhavebeenavailablehadyouchosensomeotherlocationfor yourcode.

Any projectof 1.6million linesis goingto havesometanglesnomatterhow yousliceit. Telling you to“start in init/main.c” is notmuchhelp,andevenaprintedreferencesuchasthis bookis atbestonly asnapshotconceptualmapthroughthelabyrinth;thedeveloperswill emploreyou to “just readthesource”but in aprojectthissize,findingyourwayaroundis anon-trivial problem.

Fortunately, severalexcellentopensourcepackagesexist for reverse-engineeringlargeprojects.Therearealsomany commercialtoolkitsbut startyou thinkingaboutarchitecture,I don’t wantto invite theexcusethatyoucannotafford them;thissectionwill focusonly on thosetoolswhich arefreelyavailable,with apreferencefor opensource.

Code VisualizationCodevisualizationis not aboutprettyprintoutsof controlstructuresin asourcefile. It is usefulto havesucha displayfor largeandcomplex functions,but suchcodeis notverycommonin thekernelsources(seeStyleGuide).In our everydaylife, weexpectstreetsto bestraightwith periodicintersectionsandrowsof drivewayson eitherside;we don’t buy mapsto placeour mailboxin front of thehouse,but wedo buy mapsandsurveys to seetheneighbourhoodandweusearchitecturalplansto understandanylargeconstructionproject.Evenif thoseplansarenotaccurate(they rarelyare)wecanwork out thedifferencesasweexplore.This is thepurposeof codevisualization.

Unfortunately, mostopensourceprojectshaveno functionalspecificationsandnoarchitecturalplans.Atbest,therearesomelooseguidingprinciplesasdecidedby thesteeringcommittees.Linux is nodifferent.Contraryto thebestadviceof softwareengineering,mostopensourceprojectsareinventedonthefly, madeup asprojectmembersencountereachnew designproblem.Opensourceteamsmoveorganicallywith no masterplan,andoverall, themethodworksverywell ... unlessyouarrivedlate.

4


I’m not advocatingdesignprocessbethrustuponopensource;not only doestheopensource“massivelyparallelsolutionspacerandomwalk” work, but Apache,Linux andmany otherlargeprojectsprovethatit worksverywell, asgoodor betterthanthetraditionalsoftwaremethodologies.I’m only suggestingthat,for any non-trivial changes,we needto understandthecurrentarchitecture,andthis is especiallyimportantin a fast-moving projectlike Linux. Dueto thegreatpopularityof theO/S,wehave limitedcontactwith coredevelopers,anda rapidpaceof kerneldevelopmentrendersexistingdocumentsobsoletefasterthanthey arewritten.Evenfor theseasonedteammember, theescalatingpaceandscopeof Linux developmentensures,whateverconceptualmodelthey mayhavehadwhenthey started,thereality is probablyverydifferent,andthereis no onewe can“call beforewe dig”: On-the-flyreverseengineeringtechniquesarepracticallyessential.

Reverseengineeringis theprocessof deriving a mapdirectly from theterritory usinganautomatedsystemto gatherrelationaldata,andagrapheditorto helpdistill thejumbleof call graphsandmisinformationof directorystructuresinto comprehensibleinteractionsof well-definedsubsystems.While commonin object-orientedprogramming,thepracticeis relatively unknown in theopensourcecommunity:Most projectshaveprocessesfor moving conceptualconsensusinto sourcecode,but therearevirtually no reverserepairpracticesensuringdocumentsmatchthesourcereality.

Rigi Visual Software Under standing Tool

Rigi is a codevisualizationsuitefrom theUniversityof Victoria in British Columbia.Thepackagehasbeenportedto Linux andalsorunson SunSPARCstations(SunOS),IBM RISCSystem6000(AIX)workstations,andon Windows95/NT.Rigi canextractandabstractstructuralinformationfrom thesourcecodeanddisplayresultsin a varietyof high-level views; thesystemis bundledwith parsersfor C,C++ andCOBOL(aY2K-boomlegacy?)andcanbeextendedusingtheTcl interface.

Rigi includestwo maincomponents,acommand-linesourceparsingrigiparse for creatingtheRigiStandardFormat(RSF)datatupples,andtheMotif-basedregieditdiagrameditorfor thegraphmanipulations,analysisandreportgeneration.Thekit alsoincludestoolsfor massagingraw dataproducedby theparserto removeredundantinformationandfor generatinghypertext displays

Theregieditdisplaywindow initially shows theprojectasaninvertedtreewhereleafnodescorrespondto functionsandstructures,andintermediatenodesareassignedaspreliminarysubsystemboundaries.Selectinga subsystemwill show containedelementswith thecall andreferencearcs.You thenusetheeditorto alterthis projectionandre-groupnodesto disclosetheunderlyingstructureof thesoftware.

While it is easyenoughto manipulatesourcecodegraphsusingrigiedit , populatingthegraphwith therelationaldatagleenedfrom sourcecodeis not sostraightforward.For anapplicationthesizeandcomplexity of Linux, this requiresconsiderablemassagingof databeforeyou cangeta meaningfulresult;thebuilt-in RCL scriptinglanguageallowsyou to automatemuchof theprocess,but creatingausefulmapof theentireLinux kernelwill requiresomeeffort andexperiment.

To preparethedatabasefor theeditor, thebasicprocedureis

5


1. Createa shellscriptwrapperaroundgccto substitutethepreprocessoroutputandfor all thecompilerinvokationsin theLinux makefiles.

2. RecompileLinux settingtheCC to yourscriptwrapper. Thiswill createcorrespondingregiparseoutputfiles for everyC-languagefile.

3. After compilation,theresultscanbecollectedtogether, sortedandfiltered,or processedintohypertext information(for a hypertext navigation)

4. Onceloadedinto theeditor, thebuilt-in scriptingsystemcanalsobeusedto filter thedatasetforredundantor misleadinglinks beforedisplayingtheprojectgraph.

Theprocessdoesnot requirechangesto theLinux sourcetreeandwill only mapthoseoptionswhichwereselectedin thekernel.config file.

It is worth theeffort. Onceloadedinto thegrapheditor, you havea verypowerful tool for distilling thesourcefiles into manageablebits.For example,youcangenerateforwarddependency mapsout frommain() or any otherfunctionto seethecall or referencescopeof thatitem.

A veryusefulfeatureof Rigi is theability to reframeany datastructureinto anobjectorientedview: Thestructureis portrayedastheattributesof avirtual classwith thefunctionsinvolving this structureasitsmethods.Thewholevirtual classdefinitionis groupedasa singlesubsystemandfolding this backintotheoverallgraphwill allow you to projecthigher-orderforwarddependency andinteractionsmaps.Thisprocesscanberepeatedto reducetheentirekernelinto thesubsystemslistedin Part II.

For yourdocumentationandpresentationpurposes,Rigi will alsoproducegraphview screendumpswhich canbemassagedthroughpostscripttoolssuchasgv or xv or includedin DocBookfiles;Rigi’soutputmaybesomewhatrustic,but theprogrammorethancompensatesfor aestheticsby beinga solidpieceof technicalwizardry.

Thepackagealsoincludesscripteddemosof theabstractionfeaturesusinga simpleapplication,a raytracerandaportionof theIBM SQL/DSsources.Thelatteris a dramaticdemonstrationof distilling anightmarishsoupof a flat call graphinto anelegantandinstantlycomprehensibledisplay. Otherexamplesof applyingRigi to very largeprojectsanda collectionof researchpaperscanbefoundon theRigi Homepage(http://www.rigi.csc.uvic.ca)at theUniversityof Victoria.

PBS: The Por table Bookshelf

Conceptuallyandfunctionallysimilar to Rigi, thePortableBook Shelf(PBS)from Ric Holt’sgroupattheUniversityof Waterloois anotherreverse-engineeringSwissArmy Knife for gatheringandpresentingstructuralinformationfrom a largesoftwareproject.PBShasalsobeenportedto Solaris,Linux andNT andcanbeobtainedfrom thePBSHomepage(http://www.turing.toronto.edu/pbs/index.html) at theUniversityof Toronto;thissitealsosportsapre-built demo(anddownloadabledatasets)usingthe2.0Linux kernelaswell asmany excellentsoftwarearchitectureresearchdocumentsdescribingPBSasspecificallyappliedto Linux.

6


(sidebar)Do weneed reverseengineering?

Do weneedelaborateschemesto tell uswhatis alreadyoutlinedin otherkerneldocumentationwithout thebenefitof complicatedtoolslikeRigi or PBS?Yes,otherdocumentationdoesdivide thekernelinto thesesamesubsystems,andweevendependedon thatknowledgein distilling ourconcretemodel,but whatis mostinterestingis wherewediffer: Our empiricalmodelshowsmany interactionsanddependenciesnot reportedinotherdocumentsor evenin thesourcecodecomments.By appealingto reality, wediscovernon-obvious(or forgotten)interactionssuchasthekmoddependenciesbetweenthenetwork subsystemandIPC.

Thisshouldnot imply ourempiricalview is any lessproneto omissionsor errors;our toolscananddo produceartifactswhich mustbeverifiedin thesourcecodelogic, andbecausethesetoolswork on thekernelbeingbuilt (usingourcurrent.config file) we cannotbecertainotherconfigurationswill not includeotherunseenrelationships.Whatwe cando isgaina moreaccuratemapof whatis known andto illustrateRichardFeynman’s famousremark“"...it doesn’t matterhow beautifulyour theoryis, it doesn’t matterhow smartyouare– if it doesn’t agreewith experiment,it’swrong.”

Sour ce Code NavigationOnceyouunderstandkernelstructure,andwhile youdo youractualcoding,you needto managethattangleof symbolsandfunctioncalls.Soucecodenavigationtoolsgiveyouvariousmethodsfor hypertextviewsof sourcefiles; you needto know whereyouaregoing,but for steppingthroughstaticsourcestructures,theseproductsareessential

Thesetoolscanbedividedinto two camps,theHTML browser-basedsystemsandthectagseditor-extensionsystems.Eachmethodhasits advantagesanddisadvantages,but all havethecommonobjectiveof point-and-clicknavigationthroughcross-referencesbetweenmany sourcefilesspreadovermany directories.

The Lin ux Sour ce Navigator

Oneof thefirst systemsto renderLinux in hypertext is a loneMetalabUNC webpagecalledsimply “theLinux SourceNavigator”. Thereareno mentionsof authorship,availablesourcecodeor any explanationwhy thissiteceasedto follow Linux at someunspecifiedincarnationof the2.0kernel;theonly thingsIcansayaboutthetheLinux SourceNavigator(http://metalab.unc.edu/navigator-bin/navigator.cgi?)arethatit is fastandthatit probablyalsoservedasaninspirationfor moreusefulprojectssuchasLXR.

7


LXR: The Lin ux Cross-Ref erence project

TheSourceNavigatorwastheseedof anideato useHTML asa meansto presentLinux sourcecode.LXR takesthis further, wrappingtheideainto a usefulandelegantpackagefor presentingmultipleversionsandmultiple branchesof a largeproject.

LXR is asuiteof Perl5scriptswhich,whencoupledwith asearchenginesuchasGlimpseor Ht/Dig, letsyou navigatekernelsourcesby searchingdirectories,searchingon keywords,identifiers,stringsandperlregularexpressions(regexps),andevenjust plain old hyperlinkbrowsing.ThedisplayrendersC codewith all projectidentifiersandfunctionnamesasHTML anchorslinking to thedefinitionsor to crossreferencelistson otherusesof thesymbol.

If you havea reasonablyfastInternetconnection,youcanenjoy thebenefitsof LXR throughtheLXRHomepage(http://lxr.linux.no/).Thissitesportsa full demonstrationof thesoftware,completewithGlimpse-searchcapabilities,andshows thevery latestLinux sourcesfor boththeproductionandthedevelopmentkernels.Otherdemosat theLXR siteincludesourcebrowsingof theGIMP, BSDandseveralotherlarge-scaleopensourceprojects.

Installing the LXR

If, on theotherhand,you needrapidand/orlocalaccessto thedisplay, a full installationof theLXRsearchengineis simpleto do,but theinstallationis large:Theidentifierindexesalonefor the2.3Linuxkerneltotal30Mbandtakenearlya half hourto generateon aCeleron300PC.TheLXR documentsaresparse,but not impossibleto follow if takencompletelyliterally. An overview of theinstallationgoesasfollows:

1. Install theLXR webfiles in yourwebserverdocumenttreeandsetthewebserverconfigto recognizetheCGI programs

2. createtheindex directoriesoutsideof thedocumenttree,typically in /usr/local/lxr andcreateindex subdirectoriesfor eachkernelversionto beindexed;thesekernelversionsmustalsobelistedin theversions text file in thebasedirectoryof theindexes

3. changeto theversion-specificindex directoryandrun thegenxrefprogramgiving thefull pathof thekernelsourcetreeasthetargetof thecommand.

This installsthebasicsystemto browsethesourcecodeandswitchbetweenalternateversions,plusitwill let youdo basicidentifierlookupsandto navigatethehyperlinksLXR will embedinto thesource-codedisplay

Thenext-level of LXR userequiresasearchengine,andfor this purpose,theLXR documentsrecommendtheGlimpsesearchengine,but they concedethatany text-basedsearchenginewill do justaswell; theLXR configallows for any CGI programto becalledin responseto thefull-text searchform1.Giventhisbackend,thesystemis ableto answerjustaboutany pattern-searchqueryon thekernelin amatterof seconds.

8


Using LXR

Thereareno surprisesin usingLXR. Thescreendisplayshows thecurrentpageandlistsall installedkernelversionsandarchitectures(seeFigure5-1).As youwould expect,while viewing a file for oneversionor architecture,selectingtheoptionfor analternatesourcetreewill displaythecorrespondingfile(if available)in thatnew context. ThismakesLXR anindispensabletool for makingsourcecomparisons,for example,whenadaptinginstructionsfor previouskernelsto thesituationin a moremodernsourcetree.

Figure5-1.Screenshotof an LXR page

In-Editor Sour ce BrowsingA sourcebrowsersuchasLXR maybeusefulfor largesweepsof yoursourcesor for identifierorcommentsearchingvia thead-hocperl regexps,but it is read-onlyandbrowserbased,andnot verypracticalduringyourhands-oncoding.To integratesourcenavigationinto thepracticeof writingsoftware,many programmersalreadyusecodenavigationtoolssuchasctagsin vi or etagsin EMACSbut thesetoolsarenot quiteup to indexing tagslistsaslargeandcomplex asthoseof theLinux kernel.

Global Sour ce Tags

You canthink of ShigioYamaguchi’sGlobalasctagson steroids:Globalis a portablesourcecodetagsystemfor C, YACCandJavasourcefilesandcanbeintegratedinto vi or Emacs,or usedwith a

9


stand-alonehypertext generatorto quickly locateidentifiers,objectreferencesandevenpreprocessorsymbols.

Globaltagsmaybeinvokedwithin youreditoror from thecommandline. Its featuresinclude

• Referencescopecanbelimited to directoriesallowing duplicateidentifierswithin differentsourceareas.

• Searchspacecanincludelibrary paths

• Tagfiles areplatformindependentandcanbemovedor sharedfreely betweendifferentarchitectures(for example,tagfilesmaybesharedvia NFS)

• An optionalcompactformatcanbeusedto reducediskspacerequirements

Obtaining Global

This is oneof thosegemapplicationswheredevelopmenthasall but ceasedonly becausethecurrentversionis aboutasgoodasit gets.Themostrecentversiondatesfrom 1996andis availableat theGlobalHomepage(http://wafu.netgate.net/tama/unix/global.html); it is alsoavailablethroughlargetoolkit sitessuchasMetaLab.

Building Kernel GTags

Theadvancedusesof theGlobalsystemfor generatingHTML output,leveragingtheExtendedVI andEMACSspecificfeaturesor extendingtheparsersarebeyondthescopeof thischapter;thedocumentswhich accompany Globalareunusuallycompleteandwell written.All youneedknow hereis, to theenduser, preparingacompleteGTAGS index for evena largeprojectsuchasLinux is no morecomplicatedthanthetraditionalCTAGS; all thecomplexity of theGlobalbtreeindexing is completelyanddelightfullyhidden.

Thecommand-lineuseof Globalis similarly straightforwardandwith no surprises.Usingtheincludedglobal utility from a shell,youcanquickly locateidentifiersandreferences... try this thenext timeakernelpatchleavesyouwithoutsomesymboldefined!It getsevenbetterfor theKerneldeveloper:Bycreatingseparateindexesin eachof yourlinux-VERSION directoriesanda third index in theparent/usr/src directory, searchingdonein thecontext of oneversionwill belimited to thatscopewhile asearchdonefrom therootdirectorywill locateall referencesor definitionsin all your installedversions.

There’smoregoodnews:Globaldoesnot requirea completerebuild of thedatabaseevery time thereis achangeto a few files.Thereis aninitial hit to build your tagsindex, but thereafter, tagfilescanbeupdatedincrementallyby changingto theparentdirectoryof thechangedor patchedfilesandissuingthe-i commandoption.Solidly cool.

10


XRef-Speller for EMACS

Similar in intentto GlobalundertheEMACSeditors,XRef-Spelleralsoprovidesin-editortracingofidentifiersandfunctioncallsin C, C++ andJavasourcecode.XRef extendsthisbasiclookupfunctiontoincludeotherinterestingfeaturessuchastheability to highlight asegmentof codewithin a functionandto auto-generatea new functionto factorout thatcode,or to do theoppositeandadjusta functionto beinsertedinto someotherfunction.Like theSourceCodeBrowser, XRef canalsobeusedto generatehypertext (HTML) editionsof sourcecodefiles to crossreferencesymbolsto thedefinitionsandotheruses;theirwebsiteincludesademoof thisusingthe2.2.5Linux sources(http://guma.ii.fmph.uniba.sk/linux.html).

Obtaining XRef

XRef is distributedthroughtheXRef-TechHomepage(ttp://www.xref-tech.com/)andwhiledemonstrationversionsareavailablefor download,XRef is commercialsoftwareandis licencedeitherin binaryor in source-codeform.

Using kernel-docUnderstandinga largeprogramrequiresgraspingthehigh-level designandnavigatingthetokenzoo,butit alsorequiresunderstandingthesourcecodeitself andtheintentionsof thedevelopers.While notfollowedthroughouttheLinux kernel(it wouldn’t beany fun if therewererules),many kerneldevelopers,andeveryonereadingthis book,areadoptingthekernel-docconventionsfor embeddedcomments;Tim WaughandMichaelZucchihavecontributeda perl script,scripts/kernel-doc, toparsethesecommentsinto DocBook,HTML, manpageandtext formats.Thisutility will alsolet yousingleout specificfunctionsin theC codeto beincludedor excluded.

parport_set_timeoutillustratestheman-pagelikeoutputreturnedfrom thekernel-doccommandperlscripts/kernel-doc-docbook-function parport_set_timeoutdri vers/parport/ieee1284.c.

Thekernel-doc is thecoreapplicationbehindthenew Documentation/DocBook projectheadedbyTim WaughandAlan Cox.Thisdirectoryusesasimpletemplatesystemto insertkernel-docgeneratedreferencepagesinto DocBooksources.A few “K ernelBooks” canbefoundin theDocumentation/DocBook directorywherethereis alsoaMakefile for generatingPDFandPostscriptoutput

Note: Programmer guidelines for coding kernel-doc comment blocks can be found in the sectioncalled Comment Blocks in Chapter 6.

11


parpor t_set_timeout

Name parport_set_timeout — settheinactivity timeoutfor adevice

Synopsis

long parport_set_timeout (struct pardevice * dev, long inactivity);

Arguments

dev

deviceona port

inactivity

inactivity timeout(in jif fies)

DescriptionThis setstheinactivity timeoutfor a particulardeviceon a port.Thisaffectsfunctionslikeparport_wait_peripheral. Thespecialvalue0 meansnot to call schedule while dealingwith thisdevice.

Thereturnvalueis thepreviousinactivity timeout.

Any callersof parport_wait_event for this devicearewokenup.

DescriptionThis setstheinactivity timeoutfor a particulardeviceon a port.Thisaffectsfunctionslikeparport_wait_peripheral. Thespecialvalue0 meansnot to call schedule while dealingwith thisdevice.

Thereturnvalueis thepreviousinactivity timeout.

Any callersof parport_wait_event for this devicearewokenup.

12


Development ToolsIn a perfectworld, you haveall thetime youneedto carefullycraftblemish-freesoftwarewith a carteblanchebudget.I don’t know aboutyourworld, but in mine,my clientswantthingsyesterday, andtheyaremorelikely to approveaprojectbudgetafterthey canseesomethingtangible,only rarelybefore.Somereadersmayberapturedin a labourof loveworkingon kernelcode,but othersmayneedthatextraboostof developmenttools.Fortunately, thereis help.

Developmenttoolscanbedividedinto threedomains:RAD tools,performanceevaluation,andproofingsimulators.Thefirst getyoucodingquickly, thesecondlet you verify andprobeyourprogram,andthelastgroupgiveyoua test-harnessthatavoidsthebotherandrisk of continuallyrebootingandletsyouobserveyourcodeon almostany interfaceinsteadof crossingyourfingersandinflicting it on a livemachine.

Linux offersthemostin themiddlecategory, in softwareprobesanddebuggingtools;thereis a simpleRAD kit for developingnew drivermodules,but thereis only one.Thereis alsoonly one’sandbox’applicationto run yourkernelin userspace.Youcanprobablytell from thesecountswheretheaverageprogrammerspendsmostof their time,andit is not in eitherinitial codingor in thefinal proofingof theircode.

Rapid Application Development in Kernel ProgrammingA kernelRAD tool needsto captureknowledgeof (or extract)currentkernelAPIs,structuresandfunctioncalls.It wouldknow whatis valid andin whatsequence,andit wouldbuild sourceanddocumenttemplatesfor commondesignpatterns.In actualpractice,however, youarenot likely to createmany new kernelservicesin yourkernelprogrammingcareer, andyouaremuchmorelikely to spendmostof your time usingsourcecodenavigationtoolssuchasGlobal.Thisprobablyexplainswhy therearesofew (exactlyone)kerneldevelopmentkits: Thereis no demand.It mayalsosimply beanapplicationwho’s timehasnot yet come.

Lin ux Driver Development Kit

Let’s supposeyou havesomenew pieceof hardwarewhichwill sit onsomeI/O or network portawaitingsomehandshakesequenceof events;this is thescenarioof theLinux DriverDevelopmentKit. A projectof Linux Labs,theLDDK simplifiescreatingdevicedriversby generatingthebasicmodulesourcecodefrom a a driverdescriptionfile andcorrespondingcode-templates.

Thetoolkit currentlycontainsonly oneprogram,thecommand-linebasedcompilerfor theirDriverDefinition Language(DDL); othertoolssuchasa GUI anda languageeditorareplanned,but at thetimeof writing, therecommendedpathis to just useSWIG (seebelow) to addwidgetaccessto thecompiler.Givenyourhand-craftedDDL file, theddl2c compilerwill generatea completedriverprojectsourcetree,completewith all yourmakefiles,librariesandonlinedocumentation,all readyto compileinto a

13


new kernelmodule.

While limited, theLDDK is still veryusefulfor creating(or justbootstrapping)globaldevices(singlemajornumber)or simplesubmoduleswithin a singlehardwaredevice (minor numbereddevices).Thegeneratedfilesautomaticallyinclude/proc pseudo-filereporting,/proc/sys SysctlI/O, ioctl()functionsandtheuserlibraries.Generateddriversalsoincludedefinitionsfor multi-level debuggingcodewhich canbemanipulatedvia insmod commandline parameters.

TheLDDK packageincludesexamplecodefor simpleandsubmoduledrivers,blockdevices,memorymapping,/proc functionsandaguideto passingsignalsbetweenthekernelanduserapplications.Thefull documentationanddownloadinformationfor theLDDK is availablefrom theLinux LabsSoftwarePage(http://www.llp.fu-berlin.de/pool/software/dutil/); Tcl/Tk, Perl,Phytonor Guile supportfor yourcanalsobeaddedthroughtheSWIG wrappergeneratorfrom http://www.cs.utah.edu/~beazley/SWIG/(http://www.cs.utah.edu/~beazley/SWIG/)or from theLinux LabProject.

Debugging and Kernel ProbesLinusTorvalds

"I’m afraidthatI’ve seentoomany peoplefix bugsby lookingatdebuggeroutput,andthatalmostinevitablyleadsto fixing thesymptomsratherthantheunderlyingproblems."

Torvaldsmayberight, but youcanalsohavebugswhich cannotbediscoveredthroughstaticanalysisofthesourcecode.Very often,especiallyin real-timeor multithreadedapplications,theonly way to gatherinformationaboutsystembehaviour is to watchit. Putanotherway, softwareis neverunderanyobligationto behavetheway it wasintendedto behave.

Working on theoperatingsystemitself presentssomeobviousproblemsfor debugging.Becausethekernelis whatprovidesthebasicI/O servicesfor all applicationssoftware,we arenot ableto standoutsideof its spacein thesamewaywe canstandoutsideof anotherapplication;outsideof thekernel,thereis no placeto stand.Fortunately, this is not preciselytrue:We areableto gainsomeknowledgeabouttherunningkernelthroughthekernel’sown I/O servicessuchasprintk() or sysctl(), or wecanindeedstandoutsidethekernelby standingin theI/O spaceof someotherkerneleitheron thesamemachineor on someotherbox.

14


When printk() failsSimple diagnostics and tracing can usually be done by peppering suspect codewith the kernel library printk() function. As expected, printk() provides a’printf’-like facility to log messages through syslogd . Where it works, and wherecalling the function is allowed, this technique works quite well, but as we all know,like any instrumentation, print statements inserted into code do not monitor thecode you wrote, they measure new code with print statements inserted into it. Thedifference can lead to code which may work when traced, but cease to work whenthe statements are removed.

The same warning holds true for the use of asser t() macros to add pre- andpost-conditions to your functions: While wonderful in theory, when these macrosare enabled they can fundamentally change your program. I have seen caseswhere inserting asser t() shifted stack addresses enough to prevent the bug fromoccuring and could as easily caused new instabilities to appear.

Sometimes,theonly way to understandwhy our codebehavestheway it doesis to strapit to thelabbenchandstuff it full of probesandmonitors.Thissectiondescribessomemethodsin all of thesescenarios,from thevery low-level probesto waysto enablefull symbolicdebuggingandport tapson arunningkernelimage.

Using ptrace

Unlessyouarecreatingcustomdebuggingtoolsor extendingthefacilitiesof theexisting tools,you areunlikely to work directly with this function.ptrace() is thestandardunix basisof all breakpointdebuggingandis usedto manipulatetheruntimestateof aprocessthroughtheprotocolof signalsdefinedin theheaderfiles.For example,a controllingprocesscanattachanotherusingthePTRACE_ATTACH requestandinitiatea singleinstructionstepwith PTRACE_SINGLESTEP or manipulatedatawithin thechild processor changetheregistervalueswithin a stoppedprocess.

Most Linux distributionsincludeman-pagedocumentationfor theptracesystemcall, but for the2.4kernelsupport,you will needto find themostrecenteditionof thispage.� sys/ptrace.h � > definestheinterfaceto all theprocesstraceandmanipulationsfunctionslocatedinLinux/arch/ARCH/kernel/ptrace.c.

Subterfugue(http://subterfugue.org/) is a relatedopensourceprojectwhichprovidesamobject-orientedscriptableinterfaceto ptrace() usingextensiblePythonscripts.Thiscanbeusedto do strace-linemonitoringkernelcommunicationsof user-spaceprogramsor to wrapanapplicationin asecuresandbox.Subterfugueis ableto provideservicesunableto theusualstraceutility, for example,tracingthroughclonesystemcallsandfollowing job control.

15


Debugging over the Serial Console

At long last,thereis a goodexcuseto dustoff thatVT52 in thecloset:New to the2.4kernel,a serialconsolecanbeenabledto monitorthesystemmessagesandmanipulatethesystemusinga simpleterminalor terminalemulatorprogram.With theserialconsoleenabled,youcanalsoissueSysRQcommands:SendingaBREAK followedby enterwill enabletheserialSysRQandprint a shortcommandsummary;to executea SysRQcommand,sendBREAK againfollowedby thecommandkey.

TheserialconsoleSysRQfacilitiesarealsoavailableasapatchfor the2.2serieskernalfrom Miquel vanSmoorenburg’sFTPsite.(ftp://ftp.cistron.nl/pub/people/miquels/kernel/v2.2/)

Using the GNU Symbolic Debugger (gdb)

Ratherthanmanipulatingthekerneldirectly throughptrace(), if you mustmonitoror correcta runningkernelimage,youwill probablydo this usinggdb. Thegoodnews is theGNU symbolicdebuggercaneasilymonitora runningLinux kernel;thebadnews is theaccessis read-only;youcannotsetbreakpointsor modify values.

To prepthekernelfor debugging,

1. Add debuggingsymbolsto yourobjectfilesby manuallyinsertingthe-g compileroptionintoarch/i368/Makefile.

2. Enablethe/proc filesystemin thekerneloptions;this will beyourwindow into thekernel.

3. Issuea makecleanandthenrebuild andinstall thenew kernelimage

With thesechanges,thenew kernelcanbemonitoredthroughthe/proc/kcore memoryimagepseudo-fileusingthecommand

gdb vmlinux -core /proc/kcore

If you needmorecontrolover theruntimestateor you needto monitora kernelthatis goingto crashandburn (takingyourdebuggerwith it), gdb canberun from a secondmachineandconnectedto your testbox throughaserialport; to facilitatethis connectionandto enableyou to attachto thekernelasearlyinthebootprocessaspossible,SiliconGraphicsoffersa kernelpatchwhichaddsthegdboptionto theLILO prompt;this patchwill alsoaddtheoptionto defineCONFIG_GDB in thekernelconfigsothefeaturecanbeturnedoff whenno longerrequired.TheSGI kGDB patchcanbedownloadedfrom thekGDB ProjectHomepage(http://oss.sgi.com/projects/kgdb/).

Built-in Kernel Debugger

16


SGI peoplejust don’t know whento quit. If yourkernelis amongthelist of supportedversions,they alsooffer theKDB patch(see oss.sgi.com(http://oss.sgi.com/projects/kdb/))whichwill addconfigurationoptionsfor CONFIG_KDB andCONFIG_KDB_FRAMEPTR. Whenenabled,theseoptionsprovidesymbolicdebuggingservicesdirectly in thekernel,without theneedfor a remotegdb process;theKDB patchkitalsoincludesaDocumentation/kdb directorywith copiousreleaseandusageinformation.

Kernel Sandbo xesWhatif yourchangesmaydamageyourhardware,or involvehandshakesequencesyouwould like to testthrougha regressiontestsuiteanda devicestub?If yourchangeskernelfit within thecapabilitiesof asimulator, it maymakesenseto testyourcodea user-spacevirtual machine,or ’sandbox’,to observe itsbehaviour. In many cases,thesametestscanbedonethrougha debuggeror evenby simply tracingloadingandunloadingyourkernelmodule,but in somecircumstances,youmaywantto verify yourcodebeforeyou inflict it on a realmachineandto proof yournew kernelin anicesafebubble.

Whenthekernelis runningin user-space,it is mucheasierto usestandarddebuggingandverificationtools.Debuggingwith gdb or profiling yourcodeusinggprof is no moreinvolvedthanany otherapplication.Otherdiagnostics,for examplebranchtestingandotherautomatedregressiontests,canalsobeapplied.

User Mode Kernel (UMK)

Theonly existing "Kernelin aSandbox"system,the"UserModeKernel"by Jeff Dike,startedout asanexperimentto provedoingsuchmadnesswasactuallypossible;it waslaterrecognizedto haverealdebuggingvalueandhassinceevolvedinto a majorprojectanda majorvoicebehindmainstreamkernelfacilitiessuchasptrace().

Whenrun in on aconsoleor in a terminalwindow, theUMK will appearjust likeany kernelboot,butwith somedevicesadjustedaroundthehostmachine.For example,duringthebootsequence,thenewkernelmayreportserialportsas

serial line 0 assigned pty /dev/ptyp7

; to connectto this runningkernel,youcaneitherlogin at thepromptin yourwindow or connectvia theserialport. In ourexamplethis wouldbeby pointingminicom or someothertermalprogramat/dev/ptyp7; whenyouexit from this session,theusermodekernelwill closethatport andopena newone,for example,on/dev/ttyp6

TheUMK is availablefor Linux 2.2but is primarily aimedat the2.3kernels;if youonly needto testloadablemodules,youcanalsofetchapre-compiledkernelimage.Thesoftwaresupportsnetworkemulation,but doesnot (yet) simulatemulti-processorsandmany hardwaredevicesarein needof aheroto codethemin. Your installationwill requiretheUMK patchesor thebinaryandaroot filesystemfor

17


thenew runtimekernel;this canbebuilt on any existing filesystemusingthescriptsprovided.Moreinformationon UMK andall thenecessaryfilescanbefetchedfrom Jeff Dike’sUML 2 website(http://www.mv.com/ipusers/karaya/uml/uml.html).

CPU Emulator s

Dependingon yourapplicationandits device requirements,it maybepossibleto bootyourexperimentalkernelwithin a ix86 (or otherarchitecture)emulatorsuchasVMWareor FreeVMWare/Bochs,althoughthestarkdivisionbetweentheoperatingspaceof suchemulatorswill severelylimit theamountofinformationyoucanextractfrom thetest;consideringtheextracostof thesoftwareandof themachinerequirementsneededto run thevirtual machine,it is probablyeasier, cheaperandmorefruitful topurchasea cheapbeater-boxandto usetheremotedebuggingtechnique.

Useful UtilitiesYou haveyournew customkernelup andrunning,it getssafelypastinitializationandnow you needtoknow if it is doingwhatit shouldandis not doingwhatit shouldn’t. In this stageof development,youmaynot needthebotherof a symbolicdebuggerto know yourprogramhasgoneastray(or to provethatit hasn’t) andin somecases,adebuggerwould begiving informationtoo low level or too specificto beuseful.

While it is truethatthereis not muchyoucando aboutthebehaviour of yourchangeswhile thekernelisbooting,mostoftenit is enoughto justwatchtheprogressof yourcodeandthestateof yourmachineusingthetraditionalmeansof theprogrammers’toolkit: Sincemostdevicesarereadilyimplementedasdynamicallyloadedmodules,therearerelatively few applicationsoutsideof innerkernelsubsystemsdevelopmentwhereplain old systemmonitortoolsarenot enoughto diagnosetheproblem... andasexpected,it is thosedeepcoreprocedureswhicharethemostintractableto identify, but let’s try to staypositive:90%of everydaykerneldiagnosiswill beobviousfrom thesystembehaviour. For thatother10%,thefollowing sectionsdescribesomeof theavailableoptionsavailableto Linux programmers.

System and Process Monitor sAssumingyourmodifiedkernelwill bootfarenoughto beaccessible,andalsoassumingyourchangesareamiableto beingtrackedby theusualsystemmonitortools.Theseincludethenormalarsenalof unixcommandssuchastop, fr eeandthedirectreadingof the/proc pseudofiles theseutilities reference.Very recently, otherstandardunix toolsfor processaccounting,programssuchassar, havebeenportedto Linux andwill generallygiveamoreaccuratepictureof whatthemachinewasdoingwhile it ranyourcode.

18


Anal yzing Core DumpsIt ranfor hours,theprocessmonitorsall lookednormalandthen... ooops.Thereareseveralmeansavailableto deducethefinal stateof thekernelafteranoops,rangingin their diagnosticdetailfromshowing only wheretheprogramfailed,to beingableto recreatethestateof themachine.

At oneendof thepost-mortemspectrum,theksymoopsutility, which wasincludedin theLinux sourcetreeup until the2.2distribution,will maptheoutputof thekerneloopsto theSystem.mapfile to showthenamefo thefunctionwherethefailureoccured(SeethesectioncalledTracingfroma KernelOOPSin Chapter6. Very often,knowing thelocationof a failureis enoughto diagnosea problem,but for othersituations,thereis no substitutefor a full backtraceandsnapshotof yourmachineat thetimeof death.Inthetraditionaluser-spaceprogrammingtoolkit, this is thedomainof thegdb –corecommand;thankstothecontributionsof theLinux KernelCrashDumpprojectatSiliconGraphics,it is now alsopossibletoapplythis analysisto theLinux kernel

TheLKCD kit includeskernelpatchesanduser-spacetoolsto captureacoredumpof a failing kerneltoa dedicatedSCSIpartition,andit canalsoperformsomeanalysison thegeneratedcoredumpor toreloadthekernelstatewhenthemachinereboots.

In its simplestform, LKCD will storethecorefile snapshotto /dev/vmdump, andthen,on reboot,it willfetchthis imageandgeneratea reporton/var/log/vmdump. RPMfiles andthenecessarykernelpatchescanbedownloadedfrom theLKCD homepage(http://oss.sgi.com/projects/lkcd/).

Kernel ProfilingYourprogramdoesnot oops,andit doesn’t exhibit any grossbehavioursrecognizableduringprocessmonitoring,but it still fails to maketheproperhandshakeor just needsto go faster;this is thedomainofprofiling.

ThestockLinux kerneloffersverybasicfacilitiesfor gatheringtiming statisticson certainsubsystemssuchastheSCSIparallelinterfaceandthenetwork layer, but all suchcodeis limited in resolutionby the100Hzkernelclock.As a result,thestandardkernelprofiling is prettymuchuseless.Severalauthorshavecontributedpatchesto allow fine grainprofiling but in all cases,thesepatchesarehighlyversion-specificandarenot beavailablefor all kernels;thecodeis oftenportableto otherkernelversions,andtheauthorsarealmostalwayshelpful,but if youaretinkeringwith this stuff, bepreparedfor surprises(which is to say, don’t promisetheresultsto anyoneby any specificdeadlinedate).

Profiling with gpr ofIf you havebeenfollowing this chapterfrom thestart,youmayhavealreadyconcludedthatSiliconGraphicsis abig playerin therealmof kerneldebugging;kernelprofiling is yetanotherareamade

19


possibleby thekind contributionsfrom SGIengineers.Thegprof patchesprovidedetailedtraceinformationthroughthe/proc filesystem;oncethesystemis prepared,kernprof is usedto setthesamplingratesandto togglethedatacollectionandthis datais thenanalyzedby runninggprof on thekernelimageandthegenerateddata.

Preppingthekernelis doneby applyingtheSGI patchto thesourcetreeto collectthecall arcdata;unfortunately, dueto abug in all known editionsof GCC,includingtheEGCScompiler, gccmustalsobemodifiedusingthepatchesfoundin theEGCSpatcharchive(http://egcs.cygnus.com/ml/egcs-patches/2000-04/msg00332.html). If youareunableto modify yourcompiler, analternativefix is to removeall FASTCALL supportin thekernelsources;theSGI patchkitincludesapatchto do this removal but keepin mind thatthis will changethebehaviour of all systemswhich mightusetheFASTCALL API.

As of this writing, thereis no FTParchivefor this patchotherthanin themailing list archives.If youarelooking for it in someotherarchive,thegprof patchwaspostedto theLinux KernelMailing List(http://linuxwww.db.erau.edu/mail_archives/linux-kernel/May_99/2383.html)by Dimitris Michailidis onthe14thof May, 1999.

Lin ux Trace ToolkitKarim Yaghmour’sTraceToolkit is anotherkernelinstrumentationpackagedesignedto catalogCPUandallocationeventsdown to microsecondresolution.Likeotherinstrumented-kernelpatchsets,LTT addsatracecapabilityoptionto thestandardkernelconfigurationscriptandletsyouchoosethelevel of detailandthespecificson theeventsto bemonitored.

BecausetheLTT tracksthesystemoveraspecifictimeperiod,it is particularlyusefulfor debuggingI/Olatency andsynchronizationproblems.Generatedreportsincludeeventflow, processanalysisandraweventlogs;thesereportsareall availablein user-spacevia the/proc interfaceandcanbereadastexttablesor analyzedwith theincludedGUI reportingmodule.

Thekit consistsof thekernelinstrumentationpatches,a kernelmoduleandtracedaemonto handlethedatacollection,anda GUI reportreaderwhich will translatethetraceoutputinto a readablereport.Sourcefilesanddocumentationcanbedownloadedfrom theLinux TraceToolkit Homepage(http://www.info.polymtl.ca/~karym/trace/)

IKD Debugging Tools PatchFor theLinux 2.2.xkernels,AndreaArchangellicompileda singlepatchto bundletogetherthemostcommonkerneldebuggingfacilities.Theincludedtoolscover

• Debug kernelstackoverflows

20


• Detectsoftwarelockups• Procedureflow tracing• Setbreakpointsandsinglestep• Print-EIPonvideoram• Stackmonitor• Fine-grainprofiling

Unfortunately, all of theseservicesarehighly versiondependentandwhile thepatchmaybeportabletootherkernelversions,Andreatendsto only supporta few astheneedsarise.

Thelatestversionsof theIKD patchareavailablefrom e-mind.comthroughAndrea’sFTParea(ftp://e-mind.com/pub/andrea/ikd/)

Notes1. In 1999,theGlimpsesearchenginewaspulledfrom thefreesoftwareworld andre-releasedasa

commercialproductwith theusualrestrictionson for-profit use.Beforeyou install GlimpseanditscompanionWebGlimpse,readover thenew licenseto ensureyourusewill complywith theirrequirements.

2. While Jeff refersto his systemastheUser-ModeKernel,hemostoftenabbreviatesthis asUML(user-modelinux); I havechosentheshortform of UMK to avoid confusionwith theUniversalModelling Language

21

Documents

Chapter 5. Tools for Kernel Programmers