Real-Time Experiment Real-Time Experiment Interface System: RT-LabInterface System: RT-Lab

Calin A. Culianu and David J. ChristiniCalin A. Culianu and David J. Christini

Division of CardiologyDivision of Cardiology

Weill Medical College of Cornell UniversityWeill Medical College of Cornell University

RT-LabRT-Lab• Some Background:

In 1999 we started as some experimenters who were excited about ‘having our cake and eating it too’: Real-Time capabilities in Linux

We wrote several programs to address our experiment needs

In 2001 we were given a grant by the National Science Foundation

• What we hope for the future: The purpose of our project is to give the Open

Source community a generic Real-Time Experiment Interface System

RT-LabRT-Lab• A Generic Software Experiment Interface System Utilizing

either RTAI or RT-Linux or (preferably) both.• Possible software building blocks:

RT-Linux 3.1 on Linux Kernel 2.4.4 (http://www.rtlinux.com) Comedi 0.7.60 (http://stm.lbl.gov/comedi) Qt GUI Toolkit 2.3 (http://www.trolltech.com)

• Hardware: Commodity IA32 hardware DAQ Boards (whatever Comedi supports)

Our boards happen to be: • NI AT-MIO 16e• NI PCI-MIO 16e• Measurement Computing DAS1602-16 PCI

Why RT for experiment interface?Why RT for experiment interface?

• Many biomedical experiments have hard real-time constraints:

Cardiac arrhythmia control utilizes electrical stimuli patterns that are a function of heartbeat dynamics.

Biomechanical feedback applies electrical or mechanical stimuli to the body as a function of limb or body position.

Vestibular stimulation applies electrical stimuli behind the ear to affect the posture of a person with poor balance.

Experiment interface: Why Real-Time Linux?Experiment interface: Why Real-Time Linux?

• Possible solutions: Non Multitasking OS (DOS, etc)

Obsolete, inconvenient Specialty RTOSs

Expensive, somewhat inaccessible to scientists Engineer and implement custom solid-state hardware

Difficult and inflexible

• Best solution: Real-Time Linux built on standard drivers and API’s

Familiarity of Unix, excellent software selection, ability to get best of both worlds: Great multitasking OS with a Real-Time scheduler.

Project Goals: It would be cool if…Project Goals: It would be cool if…

• Experimenters didn’t have to “re-invent the wheel”

Certain pieces of code are probably going to be common to all experiments

• Software tools existed for building custom experiment software

Open source Modular, flexible Ability to customize for exact needs of application

History: Programs We’ve WrittenHistory: Programs We’ve Written• DAQ System (http://www.rtlab.org)

Data Acquisition with nice & pretty GUI Data analysis

• AV VA Pace (http://www.rtlab.org) Real-time feedback based on intra-cardiac ECG Signal ‘Chaos-control’ algorithms tweaked in non-Real-Time by UI

• Interval Monitor (http://www.rtlab.org) Variation on a theme: Chaos control problem UI helps illustrate difficult-to-see dynamics

Historical App I: Data AcquisitionHistorical App I: Data Acquisition• Essential for a general Software Experiment System

Historical App II: AV Pace ProgramHistorical App II: AV Pace Program

• Red tracing represents heartbeat

• Bottom graphs represent various UI controls for affecting the Real-Time algorithm used to determine characteristics of the white-tracing (the stimulus to the heart)

• White tracing represents stimulus

Some ObservationsSome Observations• Hardware needs to be abstracted

Application code needs to be sufficiently shielded from implementation details.

• Graphing is Ubiquitous Powerful yet adaptable graphing tools/libraries/classes are

essential

• Experimentalists/Scientists often don’t have the time or the skills to develop software

While scientists are quite clever people, in general the most helpful toolkits are the ones that are easiest to install

Despite this fact, this does not mean we should compromise versatility for the advanced user in favor of ease-of-use for the beginner. We believe it is possible to be both powerful and obvious at the same time.

ArchitectureArchitecture• Kernel Thread:

Runs with real-time priority Communicated with user process via shared

memory and FIFO’s

• User Thread: Responsible to presenting results of kernel thread

execution to user (via Graphs, and other visualization mechanisms)

Responsible for controlling behavior of Kernel Thread via user-specified controls/parameters

Kernel-User Kernel-User Interaction: AV PaceInteraction: AV Pace

• RT-Kernel Task and NRT-process communicate via shared memory (1MB) and FIFOs.

• Non-real-time graphing, storage, …

• User control through a Qt GUI.

More on our current architectureMore on our current architecture• “rt_process.o” – an LKM that acts as a bridge between

kernel-level Comedi drivers, RT-Linux, and Userland Intelligent understanding of Comedi Present user processes with a very generic and powerful

interface to the hardware Goal is to keep user code as hardware-neutral as possible Hot-pluggable code so that “rt_proces.o” can dynamically

change its behavior Extremely important if one hopes to keep scientific experimenters

from needing to learn kernel API Approach is to provide generic interface for defining feedback

behavior, using simple API.

rt_process.o: Approach to Feedbackrt_process.o: Approach to Feedback

• How to specify dynamic feedback control? Each application probably needs its own custom

heuristics with respect to dynamic feedback in Real-Time

AV VA Pace, for instance, uses some linear algebra tricks to determine how to prevent arrhythmias

Some algorithms are stateless from sample to sample, some are stateful

rt_process.o: Approach to Feedbackrt_process.o: Approach to Feedback

• Conclusion: We cannot easily generalize• Solution: Need to keep architecture of

rt_process.o driver itself very open• What we have so far:

The venerable C-like approach: Callback Functions!

• What we hope to add: Optional ability to specify logic in an interpreted

language

rt_process.o: Callback APIrt_process.o: Callback API• “rt_process.o”, after each scan cycle, can optionally call

user-created kernel modules, meeting a standard function definition:

typedef void (*rtfunction_t)(SharedMemStruct *, MultiSampleStruct *);

/* registers a function to be run within the rtf loop returns 0 on succes, ENOMEM or EBUSY on error. Specified function won't be run (activated) until rtp_activate_function() is called

SIDE Effect: rt_process's use count is incremented */int rtp_register_function(rtfunction_t function);/* de-registers a function that was previously registered, returns 0 on

success, EINVAL or EBUSY on error SIDE Effect: rt_process's use count is decremented */int rtp_unregister_function(rtfunction_t function);/* de-activates a function that was previously registered, returns 0 on success, EINVAL on error */int rtp_deactivate_function(rtfunction_t function);/* activates a function that was previously registered, returns 0 on

success, EINVAL on error */int rtp_activate_function(rtfunction_t function);

Going Forward - DAQ: The Next GenerationGoing Forward - DAQ: The Next Generation

Other features of our current code:Other features of our current code:• Hardware Probing/Abstracting C++ classes – smart

user classes that talk to Comedi via rt_process.o and abstract-out board capabilities.

• Graph C++ classes – powerful extensible classes to address the graphing problem.

• Hot-pluggable code so that “rt_proces.o” can dynamically change its behavior with respect to how it acquires data and how it does Real-Time feedback

• It comes with a free Data Acquisition App!

ConclusionConclusion• RT capabilities are essential for many experiments

• Many experiments have a great deal in common

• Since the above is true, we, the RT-Lab project, have begun to write a generic experiment toolkit

• Software URL: http://www.rtlab.org

RT-Lab?RT-Lab?

• We received this somewhat ominous email recently: