QNX.Jan05

8/6/2019 QNX.Jan05

1/5Reprinted from Embedded Computing Design / January 2005

Speed without the expenseThe answer lies in the very nature ofembedded systems, which are oftenmanufactured by the thousands or evenmillions of units. Even a one dollarreduction in per-unit hardware costs cansave the manufacturer a small fortune.Many of these systems are cost sensitive,where the use of multi-gigahertz processorsor a large memory array is not possible. Inthe automotive telematics and infotainmentmarket, for instance, the typical 32-bit

processor runs at about 200 MHz a farcry from the 2 GHz or faster processorsnow common in desktops and servers. In anenvironment like this, an RTOS designedto extract extremely fast (and predictable)response times from lower-end hardwareoffers a serious economic advantage.

Savings aside, the services provided byan RTOS make many computing prob-lems easier to solve, particularly whenmultiple activities compete for a systemsresources. Consider, for instance, a systemwhere users expect immediate response

to input. With an RTOS, a developer canguarantee that operations initiated by theuser will execute in preference to other

By Paul N. Leroux

RTOS versus GPOS:What is best for embedded

development?Do most embedded projects still need an RTOS?

It is a good question, given the speed of todays

high-performance processors and the

availability of real-time patches

for Linux, Windows, and

other General Purpose

Operating Systems

(GPOSs).

system activities, unless a more importantactivity must execute first (for example, anoperation that protects user safety).

Consider also a system that must satisfyQuality of Service (QoS) requirements,such as a device that presents live video.If the device depends on software forany part of its content delivery, it canexperience dropped frames at a ratethat users perceive as unacceptable.From the users perspective, the deviceis unreliable. But with an RTOS, the de-veloper can precisely control the order inwhich software processes execute, andthereby ensure that playback occurs atan appropriate and consistent media rate.

RTOSs are not fairThe need for hard real time and forOSs that enable it remains prevalentin the embedded industry. For evidence,consider recent developments in theLinux world. MontaVista, for example,has launched an open source project inan attempt to improve task preemption

in the Linux kernel. Meanwhile, a recentstudy conducted by Venture DevelopmentCorporation suggests that lack of real-time

performance is the biggest impediment toLinux adoption. The questions are:

What does an RTOS have that a GPOSdoes not?

How useful are the real-time extensionsnow available for some GPOSs?

Can such extensions provide a reason-able facsimile of RTOS performance?

Task schedulingLets begin with task scheduling. In a GPOS,

the scheduler typically uses a fairnesspolicy to dispatch threads and processesonto the CPU. Such a policy enablesthe high overall throughput required bydesktop and server applications, but offersno guarantees that high-priority, time-critical threads will execute in preferenceto lower-priority threads.

For instance, a GPOS may decay the prior-ity assigned to a high-priority thread, orotherwise dynamically adjust the priorityin the interest of fairness to other threadsin the system. A high-priority thread can,

as a consequence, be preempted by threadsof lower priority. In addition, most GPOSshave unbounded dispatch latencies: the

8/6/2019 QNX.Jan05


is an upper bound on the longest non-preemptible code path through the kernel.

In a few GPOSs, such as Linux 2.6, somedegree of preemptibility has been added tothe kernel. However, the intervals duringwhich preemption may not occur are stillmuch longer than those in a typical RTOS;the length of any such preemption intervalwill depend on the longest critical section

of any modules incorporated into thekernel (for example, networking and filesystems). Moreover, a preemptible kerneldoes not address other conditions that canimpose unbounded latencies, such as theloss of priority information that occurswhen a client invokes a driver or othersystem service.

Avoiding priority inversionEven in an RTOS, a lower-priority threadcan inadvertently prevent a higher-priority thread from accessing the CPU

a condition known aspriority inversion.Generally speaking, priority inversionoccurs when two tasks of differingpriority share a resource, and the higher-priority task cannot obtain the resourcefrom the lower-priority task. To preventthis condition from exceeding a fixedand bounded interval of time, an RTOSmay provide a choice of mechanismsincluding priority inheritance and priorityceiling emulation. We could not possiblydo justice to both mechanisms here, so letus focus on a simple example of priorityinheritance.

To begin, we first must look at theblocking that occurs from synchroniza-tion in systems, and how priority inversion

Generally speaking,

priority inversion occurs

when two tasks ofdiffering priority share a

resource, and the higher-

priority task cannot obtain

the resource from the

lower-priority task.

more threads in the system, the longer it

takes for the GPOS to schedule a threadfor execution. Any one of these factorscan cause a high-priority thread to miss itsdeadlines even on a fast CPU.

In an RTOS, on the other hand, threadsexecute in order of their priority. If ahigh-priority thread becomes ready torun, it will, within a small and boundedtime interval, take over the CPU fromany lower-priority thread that may beexecuting. Moreover, the high-prioritythread can run uninterrupted until it hasfinished what it needs to do unless, ofcourse, it is preempted by an even higher-priority thread. This approach, known aspriority-based preemptive scheduling,allows high-priority threads to meet theirdeadlines consistently, no matter howmany other threads are competing forCPU time.

Preemptible kernelFor most GPOSs, the OS kernel is notpreemptible. Consequently, a high-priorityuser thread can never preempt a kernel call,but must instead wait for the entire call to

complete even if the call was invokedby the lowest-priority process in thesystem. Moreover, all priority informationis usually lost when a driver or other systemservice, usually performed in a kernelcall, executes on behalf of a client thread.Such behavior causes unpredictabledelays and prevents critical activities fromcompleting on time.

In an RTOS, on the other hand, kerneloperations are preemptible. There are stillwindows of time in which preemption maynot occur, but in a well-designed RTOS,those intervals are extremely brief, oftenon the order of hundreds of nanoseconds.Moreover, the RTOS will impose anupper bound on how long preemptionis held off and interrupts disabled; thisallows developers to ascertain worst-caselatencies.

To achieve this goal, the RTOS kernelmust be simple and as elegant as possible.Only services with a short execution pathshould be included in the kernel itself. Anyoperations that require significant work

(for instance, process loading) must beassigned to external processes or threads.Such an approach helps ensure that there

8/6/2019 QNX.Jan05


can occur as a result. Let us say two jobsare running, and that Job 1 has the higherpriority. If Job 1 is ready to execute, butmust wait for Job 2 to complete an activity,we have blocking. This blocking may occuras a result of synchronization waitingfor a shared resource controlled by a lockor a semaphore or as a result of requestinga service.

The blocking allows Job 2 to run until thecondition that Job 1 is waiting for occurs(for instance, Job 2 unlocks the resourcethat both jobs share). At that point, Job1 gets to execute. The total time that Job1 must wait may vary, with a minimum,average, and maximum time. This intervalis known as the blocking factor. If Job 1is to meet any of its timeliness constraints,this factor cannot vary according to anyparameter, such as the number of threadsor an input into the system. In other words,the blocking factor must be bounded.

Now let us introduce a third job thathas a higher priority than Job 2 but alower priority than Job 1 (Figure 1). IfJob 3 becomes ready to run while Job 2is executing, it will preempt Job 2, andJob 2 will not be able to run again untilJob 3 blocks or completes. This will, ofcourse, increase the blocking factor of Job1; that is, it will further delay Job 1 fromexecuting. The total delay introduced bythe preemption is a priority inversion.

In fact, multiple jobs can preempt Job 2 in

this way, resulting in an effect known aschain blocking. Under these circumstances,Job 2 might be preempted for an indefiniteperiod of time, yielding an unboundedpriority inversion, causing Job 1 to failto meet any of its timeliness constraints.This is wherepriority inheritance comesin. If we return to our scenario and makeJob 2 run at the priority of Job 1 during thesynchronization period, then Job 3 will notbe able to preempt Job 2, and the resultingpriority inversion is avoided (Figure 2).

Dueling kernelsGPOSs Linux, Windows, and variousflavors of UNIX typically lack themechanisms we have just discussed.Nonetheless, vendors have developeda number of real-time extensions andpatches in an attempt to fill the gap. Thereis, for example, the dual-kernel approach,in which the GPOS runs as a task on top ofa dedicated real-time kernel (Figure 3).

Any tasks that require deterministicscheduling run in this kernel, but at a higherpriority than the GPOS kernel. These tasks

can thus preempt the GPOS whenever theyneed to execute and will yield the CPU tothe GPOS only when their work is done.

Figure 1

Figure 2

Figure 3

8/6/2019 QNX.Jan05


In Linux, for example, the driver andVirtual File System (VFS) frameworkswould effectively have to be rewrittenalong with any device drivers and filesystems employing them. Withoutsuch modifications, real-time taskscould experience unpredictable delayswhen blocked on a service. As a furtherproblem, most existing Linux drivers arenot preemptible. To ensure predictability,

programmers would also have to insertpreemption points into every driver inthe system.

All this points to the real difficulty, andimmense scope, of modifying a GPOSso it is capable of supporting real-timebehavior. However, this is not a matter ofRTOS good, GPOS bad. GPOSs such asLinux, Windows XP, and UNIX all servetheir intended purposes extremely well.They only fall short when they are forcedinto deterministic environments they

were not designed for, such as thosefound in automotive telematics systems,medical instruments, and continuousmedia applications.

What about an RTOS?Still, there are undoubted benefits tousing a GPOS, such as support for widelyused APIs, and in the case of Linux, theopen source model. With open source, adeveloper can customize OS componentsfor application-specific demands and saveconsiderable time troubleshooting. TheRTOS vendor cannot afford to ignore these

benefits. Extensive support for POSIX

APIs the same APIs used by Linux andUNIX is an important first step. So isproviding well-documented source andcustomization kits that address the specificneeds and design challenges of embeddeddevelopers.

The architecture of the RTOS also comesinto play. An RTOS based on a microkerneldesign, for instance, can make the job of

OS customization fundamentally easierto achieve. In a microkernel RTOS, onlya small core of fundamental OS services(such as signals, timers, and scheduling)reside in the kernel itself. All othercomponents (such as drivers, file systems,protocol stacks, and applications) runoutside the kernel as separate, memory-protected processes (Figure 4).

As a result, developing custom drivers andother application-specific OS extensionsdoes not require specialized kernel

debuggers or kernel gurus. In fact, as user-space programs, such extensions becomeas easy to develop as regular applications,since they can be debugged with standardsource-level tools and techniques.

For instance, if a device driver attemptsto access memory outside its processcontainer, the OS can identify the processresponsible, indicate the location of thefault, and create a process dump fileviewable with source-level debuggingtools. The dump file can include all theinformation the debugger needs to identify

the source line that caused the problem,

Figure 4

Unfortunately, tasks running in the real-time kernel can make only limited use ofexisting system services in the GPOS filesystems, networking, and so on. In fact, ifa real-time task calls out to the GPOS forany service, it will be subject to the samepreemption problems that prohibit GPOSprocesses from behaving deterministically.As a result, new drivers and systemservices must be created specifically for

the real-time kernel even when equivalentservices already exist for the GPOS.

Also, tasks running in the real-time kerneldo not benefit from the robust MemoryManagement Unit (MMU) protectedenvironment that most GPOSs provide forregular, non-realtime processes. Instead,they run unprotected in kernel space.Consequently a real-time task that con-tains a common coding error, such asa corrupt C pointer, can easily cause afatal kernel fault. To complicate matters,different implementations of the dual-

kernel approach use different APIs. Inmost cases, services written for the GPOScannot be easily ported to the real-timekernel, and tasks written for one vendorsreal-time extensions may not run onanothers real-time extensions.

Modified GPOSsRather than use a second kernel, otherapproaches modify the GPOS itself,such as by adding high-resolution timersor a modified process scheduler. Suchapproaches have merit, since they allowdevelopers to use a standard kernel(albeit with proprietary patches) andprogramming model. Moreover, theyhelp address the requirements of reactive,event-driven systems.

Unfortunately, such low-latency patchesdo not address the complexity of mostreal-time environments, where real-timetasks span larger time intervals and havemore dependencies on system servicesand other processes than do tasks in asimple event-driven system. For instance,in systems where real-time tasks depend

on services such as device drivers or filesystems, the problem of priority inversionwould have to be addressed.

8/6/2019 QNX.Jan05


A strategic decisionAn RTOS can help make complexapplications both predictable and reliable.In fact, the predictability made possibleby an RTOS adds a form of reliabilitythat cannot be achieved with a GPOS (if asystem based on a GPOS does not behavecorrectly due to incorrect timing behavior,then we can justifiably say that the systemis unreliable). Still, choosing the right

RTOS can itself be a complex task. Theunderlying architecture of an RTOS isan important criterion, but so are otherfactors.

Consider Internet support. Does theRTOS support an up-to-date suite ofprotocol stacks such as IPv4, IPv6, IPsec,SCTP, and IP filtering with NAT? Andwhat about scalability? Does the RTOSsupport a limited number of processes, ordoes it allow hundreds or even thousandsof processes to run concurrently? And

does it provide support for distributed orsymmetric multiprocessing?

GUI considerationsGraphical User Interfaces (GUIs) arebecoming increasingly common in

embedded systems, and those interfacesare becoming increasingly sophisticated.Consequently, does the RTOS supportprimitive graphics libraries, or does itprovide an embeddable windowing systemthat supports 3D rendering, multi-layerinterfaces, and other advanced graphics?Can you customize the GUIs look-and-feel? Can the GUI display and inputmultiple languages simultaneously? And

does the GUI support an embeddableweb browser? The browser should havea scalable footprint, and be capableof rendering web pages on very smallscreens. It should also support currentstandards such as HTML 4.01, XHTML1.1, SSL 3.0, and WML 1.3.

Tool considerationsOn the tools side, does the RTOS vendoroffer diagnostic tools for tasks such astrace analysis, memory analysis, applica-tion profiling, and code coverage? And

what of the development environment? Isit based on an open platform like Eclipse,which lets you readily plug in third-party tools for modeling, version control,and so on? Or is it based on proprietarytechnology?

On one point, there is no question. TheRTOS can play a key role in determininghow reliable your system will be, howwell it will perform, and how easily it willsupport new or enhanced functionality. Andit can support many of the rich servicestraditionally associated with GPOSs,

but implemented in a way to address thesevere processing and memory restraintsof embedded systems.

Paul Leroux is a

Technology Analystat QNX Software

Systems, where he

has served in variousroles since 1990. His

areas of focus include

OS architecture, highavailability systems,

and integrated

development environments.

For more information, contact Paul at:

QNX Software Systems Ltd.

175 Terence Matthews Crescent

Ottawa, Ontario, Canada K2M 1W8

Tel: 613-591-0931 Fax: 613-591-3579

E-mail: [email protected]

Website: www.qnx.com

along with diagnostic information such asthe contents of data items and a history offunction calls.

This architecture also provides superiorfault isolation. If a driver, protocol stack,or other system service fails, it can do sowithout corrupting other services or theOS kernel. In fact, software watchdogs cancontinuously monitor for such events and

restart the offending service dynamically,without resetting the entire system orinvolving the user in any way. Likewise,drivers and other services can be stopped,started, or upgraded dynamically, againwithout a system shutdown.

Documents

QNX.Jan05