RT-LINUX

Introduction.

RT Linuxis ahard real timeRTOSmicrokernelthat runs the entireLinuxoperating systemas fullypreemptiveprocess. It is one of the hard real-time variants of Linux, among several, that makes it possible to control robots, data acquisition systems, manufacturing plants, and other time-sensitive instruments and machines.It was developed by Victor Yodaiken, Michael Barabanov, Cort Dougan and others at theNew Mexico Institute of Mining and Technologyand then as a commercial product atFSMLabs.Wind River Systemsacquired FSMLabs embedded technology in February 2007 and made a version available as Wind River Real-Time Core forWind River Linux. As of August 2011, Wind River has discontinued the Wind River Real-Time Core product line, effectively ending commercial support for the RTLinux product.The key RTLinux design objective was to add hard real-time capabilities to a commodity operating system to facilitate the development of complex control programs with both capabilities.For example, one might want to develop a real-time motor controller that used a commodity database and exported a web operator interface. Instead of attempting to build a single operating system that could support real-time and non-real-time capabilities, RTLinux was designed to share a computing device between a real-time and non-real-time operating system so that (1) the real-time operating system could never be blocked from execution by the non-real-time operating system and (2) components running in the two different environments could easily share data. As the name implies RTLinux was the first computer designed to use Linux as the non-real-time systembut it eventually evolved so that the RTCore real-time kernel could run with either Linux orBSD UNIX.Multi-Environment Real-Time(MERT) was the first example of a real-time operating system coexisting with a UNIX system. MERT relied on traditional virtualization techniques: the real-time kernel was thehostoperating system (orhypervisor) and Bell Systems UNIX was theguest. RTLinux was an attempt to update the MERT concept to the PC era and commodity hardware. It was also an attempt to also overcome the performance limits of MERT, particularly the overhead introduced by virtualization.The technique used was to only virtualize the guest interrupt control. This method allowed the real-time kernel to convert the guest operating system into a system that was completely preemptible but that could still directly control, for example, storage devices. In particular, standard drivers for the guest worked without source modification although they needed to be recompiled to use the virtualization "hooks". See alsoparavirtualization. The UNIX "pipe" was adapted to permit real-time and non-real-time programs to communicate although other methods such as shared memory were also added.From the programmer's point of view, RTLinux originally looked like a small threaded environment for real-time tasks plus the standard Linux environment for everything else. The real-time operating system was implemented as aloadable kernel modulewhich began by virtualizing guest interrupt control and then started a real-time scheduler. Tasks were assigned static priorities and scheduling was originally purely priority driven. The guest operating system was incorporated as the lowest priority task and essentially acted as the idle task for the real-time system. Real-time tasks ran in kernel mode. Later development of RTLinux adopted thePOSIX threadsapplication programming interface (API) and then permitted creation of threads in user mode with real-time threads running inside guest processes. In multiprocessor environments threads were locked to processor cores and it was possible to prevent the guest thread from running on designated core (effectively reserving cores for only real-time processing).RTLinux provides the capability of running special real-time tasks and interrupt handlers on the same machine as standard Linux. These tasks and handlers execute when they need to execute no matter what Linux is doing. The worst case time between the moment a hardware interrupt is detected by the processor and the moment an interrupt handler starts to execute is under 15 microseconds on RTLinux running on a generic x86 (circa 2000). A RTLinux periodic task runs within 25 microseconds of its scheduled time on the same hardware. These times are hardware limited, and as hardware improves RTLinux will also improve. Standard Linux has excellent average performance and can even provide millisecond level scheduling precision for tasks using the POSIX soft real-time capabilities. Standard Linux is not, however, designed to provide sub-millisecond precision and reliable timing guarantees. RTLinux was based on a lightweight virtual machine where the Linux "guest" was given a virtualized interrupt controller and timer, and all other hardware access was direct. From the point of view of the real-time "host", the Linux kernel is a thread. Interrupts needed for deterministic processing are processed by the real-time core, while other interrupts are forwarded to Linux, which runs at a lower priority than real-time threads. Linux drivers handle almost allI/O. First-In-First-Out pipes (FIFOs) or shared memory can be used to share data between the operating system and RTLinux.RTLinux is structured as a small core component and a set of optional components. The core component permits installation of very low latency interrupt handlers that cannot be delayed or preempted by Linux itself and some low level synchronization and interrupt control routines. This core component has been extended to support SMP and at the same time it has been simplified by removing some functionality that can be provided outside the core.1. Functionality:The majority of RTLinux functionality is in a collection of loadable kernel modules that provide optional services and levels of abstraction. These modules include:1) rtl sched a priority scheduler that supports both a "lite POSIX" interface described below and the original V1 RTLinux API.2) rtl time which controls the processor clocks and exports an abstract interface for connecting handlers to clocks.3) rtl posixio supports POSIX style read/write/open interface to device drivers.4) rtl fifo connects RT tasks and interrupt handlers to Linux processes through a device layer so that Linux processes can read/write to RT components.5) semaphore is a contributed package by Jerry Epplin which gives RT tasks blocking semaphores.6) POSIX mutex support is planned to be available in the next minor version update of RTLinux.7) mbuff is a contributed package written by Tomasz Motylewski for providing shared memory between RT components and Linux processes.

2. Kernel module:

RTLinux is in fact a Linuxkernel module. It is the same type of module that Linux uses for drivers, file systems, and so on. The main difference between RTLinux module and an ordinary Linux module is that RTLinux module calls functions, which are offered by RTLinux kernel whereas ordinary module uses only Linux kernel functions. The source code of a simple Linux module is given below. It contains two functions: init_module, which is called when the module is inserted to the kernel byinsmodormodprobecommand, and cleanup module, which is called before module is removed by rmmod.

3. Architecture:

Fig 1: Architecture of RTLinux.Standard time sharing OS and hard real time executive running on same machine Kernel provided with the emulation of Interrupt control H/W. RT tasks run at kernel privilege level to provide direct H/W access. RTLinux runs Linux kernel for booting, device drivers, networking, FS, process control and loadable kernel modules. Linux kernel runs as a low priority task on the RT kernel hence making sure it cannot preempt any RT task.RT task allocated fixed memory for data and code.RT tasks cannot use Linux system calls , directly call routines or access ordinary data structures in Linux Kernel. RTProcesses without memory protection features (RTLinux Pro has some PSDD now). The RT kernel is actually patched over the Linux kernel and then recompiled to work as a RTLinux system. Completely configurable RTLinux kernel.VM layer only emulates the Interrupt Control. The 0 level OS does not provide any basic services that can be provided by Linux - Only RT services - No primitives for process creation, switching or MM. Uses software stack for switching and not the expensive H/W switching. RTKernel is not preemptable.

Objectives

The key RTLinux design objective is that the system should be transparent, modular, and extensible. Transparency means that there are no unopenable black boxes and the cost of any operation should be determinable. Modularity means that it is possible to omit functionality and the expense of that functionality if it is not needed. The base RTLinux system supports high speed interrupt handling and no more. And extensibility means that programmers should be able to add modules and tailor the system to their requirements. It has simple priority scheduler that can be easily replaced by schedulers more suited to the needs of some specific application. When developing RTLinux, it was designed to maximize the advantage we get from having Linux and its powerful capabilities available.

Detailed Problem Definition

There are numerous real-time operating systems available today. What was missing, however, is an open, standard, supported, efficient, and inexpensive multitasking system with hard real-time capabilities. Many UNIX systems meet the first three requirements. Linux 1, a relatively recent free UNIX-like OS, originated by Linus Torvalds, features excellent stability, efficiency, source code availability, not restrictive license, and substantial user base. Source code availability is essential for verification of the system correctness, adaptation to specific problems, and mere bug fixing. Linux can run on the most widely available computers: IBM PCs and compatible ma-chines (most PC hardware is supported). Since many computer and engineering labs have computers of this type, it is possible for them to use Linux without any further investment.Linux has all features of a modern UNIX system: several X Window System implementations, graphical user interface toolkits, networking, databases, programming languages, debuggers, and a variety of applications. It is embeddable and is able to perform efficiently using relatively small amounts of RAM and other computer resources. In short, Linux has the potential to make an excellent development platform for a wide variety of applications, including those involving real-time processing. However, Linux has several problems preventing it from being used as a hard real-time OS, most notably the fact that interrupts are often disabled during the course of execution of the kernel. Other problems include time-sharing scheduling, virtual memory system timing unpredictability, and lack of high-granularity timers. It turns out that using software interrupts, together with several other techniques, it is nevertheless possible to modify Linux so as to overcome these problems. The idea to use software interrupts so that a general-purpose operating system could coexist with a hard real-time system is due to Victor Yodaiken (personal communications). This thesis details how this idea and several others were applied to build a hard real-time version of Linux.Solution Methodology.

RTLinux is very much module oriented. To use RTLinux, you load modules that implement whatever RT capabilities you need. Two of the core modules are the scheduler and the module that implements RT-fos. If the services provided by these modules dont meet the requirements of the application, they can be replaced by other modules. For example, there are two alternative scheduling modules a earliest deadline rst scheduler implemented by Ismael Rippol and a rate-monotonic scheduler. Fig.2: Flow of data and control

The basic scheduler simply runs the highest priority ready RT task until that task suspends itself or until a higher priority task becomes ready.The original RTLinux scheduler used the timer in one shot mode so that it could easily handle a collection of tasks with periods that had a small common divisor. For example, if one task must run every331time units and the other runs every1027time units, there is no good choice for a timer period. In one-shot mode, the clock would be set rst to generate an interrupt after 331 time units and then reprogrammed after the interrupt to generate a second interrupt in another691time units (minus the time needed to reprogram the clock). The price we pay is that we reprogram the clock on every interrupt. For x86 generic motherboards, reprogramming the clock is relatively slow. It turns out, how- ever, that many applications dont need the generality of a one-shot timer and can avoid the expense of reprogramming. Professor Paolo Mantegazza of the Aerospace Engineering Department in Politecnico di Milano wrote a scheduler that demonstrated the utility of periodic mode and encouraged us to put it into the standard scheduler6. The current RTLinux scheduler offers both periodic and one shot modes. On SMP systems the problem gets simpler because there is an on-processor high frequency timer chip that is available to the RTL system.

Details of experimentation, analysis, modelling.

In order to measure the performance of Real-Time Linux, I have conducted several experiments. The experiments were performed on two IBM PC compatible computers running Linux version 2.0.29 and Real-Time Linux version 0.5a. The results are summarized in Table. To measure the maximum interrupt latency, an additional machine running Real-Time Linux (Machine 1) was used to send interrupt requests to the machine being tested (Machine 2) and to measure the response time of the latter (Figure 1.2).The D0 output of the parallel port of Machine 1 is connected to the ACK parallel port input of Machine 2. When the ACK input goes from a logic one to logic zero, an interrupt request is sent to the processor. To provide feedback, the D0 output of Machine 2 is connected to a parallel port input (PE) on Machine 1.

Fig.2: Measuring interrupt latencyTable 1: interrupt latency and scheduling precision for different machines.

Machine A: Intel 80486 33 MHz, ISA bus, 16 MB of RAM, Western Digital Caviar 2340 IDE hard drive, 3C509 Ethernet card.Machine B: Intel Pentium 120 MHz, PCI bus, 32 MB of RAM, EATA-DMA DPT SCSI adapter, Conner CP30200 SCSI hard drive, NE2000 Ethernet card (in an ISA slot).The measurement is performed as follows. A real-time process on Machine 1 records the current time, sends a pulse to the D0 output, and enters a busy loop waiting for the PE input to change. An interrupt request is sent to the CPU of Machine 2. From an interrupt handler (both real-time and ordinary Linux handlers were used) the output D0 is toggled. The real-time process on Machine 1 exits the busy loop, obtains the current time, and computes how long it took Machine 2 to respond. This sequence is performed periodically over a substantial interval of time. The maximum response time encountered is taken to be an estimate of the worst-case interrupt latency.As seen from Table, the interrupt handling latency in plain Linux is substantially higher than in Real-Time Linux. Moreover, it is quite possible that some device drivers that I have not used disable interrupts for longer periods, further increasing Linux interrupt processing latency.To measure scheduling precision a periodic real-time task was run. On each wake-up the current time was obtained and compared to the estimate. Maximum deviations were recorded.I found it impossible to reliably run periodic tasks as standard Linux processes, partly because Linux does not provide periodic timers facility, and also because of other problems .During all tests the system was heavily loaded with disk and network I/O operations. Although device drivers in Real-Time Linux do not disable hardware interrupts, heavy I/O does increase interrupt latency. This fact can be attributed to the DMA cycle stealing.Several stress tests have also been performed. In one of them the system has successfully performed a backup of a local system over the local network, scheduling two periodic real-time tasks each with 1 millisecond period at the same time. This experiment demonstrates that the presence of a real-time system has no adverse effect on the functioning of the Linux kernel.Overall, the results show that Real-Time Linux is a viable platform for hard real-time processing.

APPLICATIONS

The first one is that of Harald Stauss2 from the department of physiology at the Hum-boldt University in Berlin, Germany. He has implemented a system for the recording and display of hemodynamic measurements in rats. It uses an analog-to-digital converter card to acquire the signals from sensors. The card based on the MAX192BCPP chip has eight 12-bit channels that are multiplexed to one serial line connected to the serial port of the computer. The system runs under Real-Time Linux and consists of a real-time task and a user process. The real-time task polls the card and passes the received data through an RT-FIFO to the user process. The process records the data to a _le and at the same time displays it graphically in a Motif drawing widget. Mr. Stauss reports that the system is able to reliably acquire data with the total sampling rate of 3000 Hz on a i486/33MHz-based machine. On the same computer, Labtech Notebook for DOS, a commercial data acquisition program, could only provide sampling rates of less than 400 Hz.Bill Crum at New Mexico Tech has developed an embedded control and monitoring software for the Tech Sunrayce car [3]. The system specification required data collection from 70 sensors with total sampling rate of 80 Hz. The needed response times ranged from 200 to 300 milliseconds. These requirements were satisfied by using one real-time task executing with 0.0125 seconds period on a 20 MHz i386{based computer running Real-Time Linux. To facilitate the debugging of the system, Bill Crum has written a set of programs that he describes as an \engineering workbench". The programs simulate tools commonly found in electrical engineering laboratories: an oscilloscope, a logic analyzer, and a signal generator. The complex uses a data acquisition card that provides analog and digital I/O. Real-time tasks are used to communicate with the card, and Linux processes implement graphical interfaces using the Qt widget set.There are several other applications. A task running under Real-Time Linux is used for real-time communication with the Phantom, a force feedback device. This is a part of a system that creates virtual worlds. The system allows users to navigate through these worlds, to feel and manipulate objects. Dan Samber4 from Mount Sinai Medical School in New York City uses Real-Time Linux to reliably communicate with a patient monitor over a serial line for recording and display of physiological parameters.

ConclusionsBy observing various features of the RTLinux and outcomes of the various analysis and experiments yields a conclusion that the Real-Time Linux is a viable platform for hard real-time processing which involves low interrupt latency and requires more scheduling precision.