[ ] [] [ ]



               FSMLabs Lean POSIX for RTLinux[IMAGE png]

                           Victor Yodaiken1
                    Finite State Machine Labs Inc.
       Finite State Machine Labs 1999,2000. All rights reserved

Abstract:

RTLinux is a small, fast operating system, following the POSIX 1003.13
"minimal realtime operating system" standard. The key design objective
of RTLinux is to produce a peaceful resolution to an apparently
insoluable contradiction: the need for simplicity and ``bare metal"
performance in a hard realtime operating system, and requirements for
all sorts of complex software, such as networking and graphical user
interfaces in modern realtime application code. RTLinux was originally
released as a research project in 1995 and provided a simple API that
was quite succesful, but since 1999 RTLinux has been tranforming into
an an industrial strength version based on a POSIX threads API. In this
article, I'll discuss what's important about ``hard realtime", how
RTLinux works, ``why POSIX" and how we reconciled RTLinux with a
standard while keeping it small and efficient.

Introduction: RTLinux, realtime and POSIX

RTLinux is a hard realtime operating system. That is, RTLinux is
designed to support applications that have real, serious,
non-negotiable deadlines. A rocket engine emergency shutdown sequence
must complete before the rocket explodes; a servo motor controller must
send commands exactly when the motor is in position; a video editing
system must capture the next frame and not permit it to be overwritten;
a data acquisition system can't miss any samples. All these systems are
hard realtime systems and for these systems worst case timing is
critical. In non-realtime applications average or typical times are
more important. If a file system averages 100Mbytes/second in data
transfer, we don't care if it stops now and then to shuffle buffers.
Similarly, if a word processor is delayed a few milliseconds, nobody
will notice, if X-windows runs fast almost all the time, a hitch now
and then is no problem. It's even reasonable to allow a consumer video
player to drop frames now and then or to live with an occasional pop or
drop-out in a soft CD player. Systems that are realtime, but where
missing a deadline now and then is acceptable are called soft realtime
systems. But usually shutting down the rocket before it explodes or
often putting a chip on an automated assembly line into the right
socket is not good enough. A delay of a few microseconds can cause a
servo-motor to get a degree out of position and you can't compensate
for being too late once or twice by being early a few times.

Twenty years ago, applications needing hard realtime were simple and
were usually placed on dedicated, custom, isolated hardware. Modern
realtime applications must control systems such as factory floors
connected to supply databases, telescopes connected to the Internet,
cell phones generating graphical displays, routers, and telephone
switches. These applications run platforms that have non-realtime
responsibilities as well as realtime ones, must work on commodity
hardware, and need to be networked. In other words, these applications
are not simple and do not run on dedicated, custom, or isolated
hardware. Furthermore, as speed and quality requirements increase,
applications that were not realtime, change into hard realtime
applications. For example, serious use of networked video and audio
requires software that can maintain streams of data with hard realtime
guarantees. As another example, variations in network transmit times
for loosely coupled stand-alone servers on relatively slow networks has
not been a problem, but a server cluster on 10G ethernet in which there
are tightly connected distributed applications is a very different
situation and one where real-time networking capabilities become an
issue.

The problem here is that to deliver the tight worst case timing
performance needed for hard realtime, operating systems need to be
simple, small, predictable - and optimized to minimize worst case
performace. But sophisticated services and powerful applications run
best on complex operating systems that are optimized to deliver strong
average case performance. TCP network stacks, for example, use
algorithms that trade latency for throughput. Software that tries to
optimize for two contradictory goals will deliver neither. In fact, the
Mars Lander, running VxWorks had precisely this problem as a standard
non-realtime inter-process communication mechanism caused a critical
realtime process to hang up and time out. RTLinux offers an alternative
to the traditional choices of ``make a general purpose operating system
kind-of-realtime" (IRIX) and ``keep adding nonrealtime features to a
realtime operating system" (VxWorks).

Instead of trying to attempting to balance worst case time against
average case time, RTLinux decouples the realtime and non-realtime
systems. We put realtime components in a single, multithreaded,
realtime process running on a bare machine. Applications are threads
and signal handlers. This realtime process is simple, fast, predictable
and optimized to minimize worst case performance. A general purpose
operating system is run as the lowest priority thread. In RTLinux this
thread is Linux. A patented "virtual interrupt controller" prevents the
low priority thread from blocking interrupts aimed at realtime signal
handlers. Communication between realtime components and the general
purpose OS thread is designed to mimimize interference -- the realtime
components are never forced to wait for operations of the non-realtime
thread. But communication mechanisms are designed to help programmers
move all the non-realtime functionality into the non-realtime
environment.

A realtime application may consist of a periodic thread running in the
base realtime process and collaborating with control software running
under Linux and making use of, for example, an Oracle database, the KDE
desktop environment, and a Perl networking script. Linux and its
processes can never block the realtime process scheduling or event
handling and the realtime process does not need to offer virtual memory
or a journaling file system or a TCP stack as these are all available
within Linux. The classic RTLinux application consists of a periodic
thread collecting data from an A/D device and driving an D/A output,
with data flowing to a Linux process that logs results and commands
comming from a Linux process that presents the operator with buttons
and a display. Of course, the challenge with this model is to make sure
that hardware efficiency does not come at the expense of programmer
efficiency and this brings us to the API.

API development

The first widely used version of RTLinux was released in 1995. This
"V1" system was intended to run on low end x86 based computers and
provided a spartan API and programming environment. Faster machines
(and more complex applications), multiprocessing, multiple
architectures, and a much larger programmer community exposed
limitations of the original API. We realized that repairing and
extending our simple API would transform it into a mess. We also
realized that with over 100 commercial RTOS's in the market, the world
was not exactly crying out for another new RTOS API. POSIX offered
several advantages: POSIX is a real standard, not an effort to lock
customers into a proprietary API; and POSIX is widely known and well
documented. But POSIX seemed inherently slow because it incorporated
too many operations that were impossible to provide in our lean
programming environment. For example, consider how very far from fast,
simple, predictable, and efficient it is to parse a POSIX file name
though links, hardlinks, mounted file systems, and layers of
directories. Fortunately, we discovered that the POSIX 1003.13 standard
was the solution we needed.

POSIX 1003.13 provides a beautiful2 model for fitting RTLinux tasks and
interrupt handlers into the POSIX world. The POSIX standard identifies,
in section 6, a "Minimal Realtime System Profile" (PSE51) intended for
hard realtime systems like RTLinux. The 1003.13 ``minimal realtime
system" application environment consists of a single, multithreaded
POSIX process running on a bare machine. In this model, RTLinux tasks
became POSIX threads and RTLinux interrupt handlers became signal
handlers. Linux runs as the lowest priority thread. In an SMP or other
multi-processor system we decided to provide a single realtime process
on each processor.

The "rationale" given in the standards document is that "the POSIX.1c
Threads model (with all options enabled, but without a file system)
best reflected current industry practice in certain embedded realtime
areas. Instead of full file system support, basic device I/O (read,
write, open, close, control) is considered sufficient for kernels of
this size. Systems of this size frequently do not include process
isolation hardware or software; therefore, multiple processes (as
opposed to threads) may not be supported."

POSIX 1003.13 escapes from the POSIX file system by providing a
bare-bones interpretation of file path names. The standard RTLinux open
will open /dev/x for a fixed set of devices and will not support any
other path names.

That's not to say that the Pthreads API is perfect. For example, the
Pthreads API has quite clumsy handling of periodic threads, the entire
cancel system is terrible, and signals are sometimes confusing. But the
API is usable and can be made better by the addition of some POSIX
compatible extensions.

In the next section, I will explain how a RTLinux program can be
constructed using the POSIX API. But it's worth noting that there is an
alternative high level interface to RTLinux from Linux user code via an
added Linux system call. The call, rtlinux_sigaction designates some
function in the user program as a signal hander to be run in realtime
mode either periodically or in response to some interrupt (as a user
interrrupt handler). The signal handler code runs in the address space
of the process so, with appropriate use of data structures, data can be
shared between the non-realtime main process and the realtime handlers.
The rtlinux_sigaction interface provides a method for writing realtime
code within Linux processes. This method is exceptionally simple to
use, but is less flexible than the RTLinux pthreads API and suffers a
(couple of microsecond) performance penalty. For more details, see the
section on the Programmable Signal Control (PSC) modules in the RTLinux
documentation.

An example

RTLinux makes use of the Linux kernel module facility to load realtime
components. Linux boots up in non-realtime mode. To switch over to
realtime operation, we insert a collection of kernel modules providing
RTLinux functionality. The modules have been designed so that users can
select a minimal subset that supports their applications. RTLinux is
provided with a command insrtl to load the full set of modules, but the
modules are designed so that users can pick a minimal set that provides
only the services they need. The RTLinux modules include rtl (basic
services), rtl_timer (to control hardware clocks), rtl_sched (the
scheduler) and rtl_posixio (a read/write/open layer for drivers. Once
the RTLinux modules are loaded, RTLinux applications can be loaded. The
standard Linux GDB debugger can be used to debug RTLinux threads. It's
also possible to use the RTLinux system purely from user mode via a
realtime signal handler facility.

RTL Threads

In order for the realtime system to be able to use Linux, we provide
three three standard interfaces between realtime tasks and Linux. The
first is a device interface called RTfifos (illustrated below),r the
second is shared memory, and the third uses the pthread signal
mechanism to allow realtime threads to generate soft interrupts for
Linux. A simple A/D application might consist of a single periodic
thread that collects data from a A/D device every 500 microseconds and
dumps the data down a fifo for Linux to log or otherwise process.

A thread function returns an untyped pointer (in "C" a void pointer can
point to any type of data structure), and takes a single argument that
is also an untyped pointer. In our case, we don't need either. The code
below declares a POSIX timespec to specify its period, and a structure
D which will hold data from the A/D device. The thread calls the POSIX
clock_gettime to read the current time from the RTL scheduling clock
into the timespec structure. Each cycle, the thread collects data from
the A/D device, writes the data down a fifo to a Linux process, adds
its period in nano-seconds to the timespec structure and calls the
POSIX clock_nanosleep to sleep until that time.

void *my_code(void *arg){ // argument is not used in this example.
     struct timespec t;
     struct mydata D;
     clock_gettime(CLOCK_RTL_SCHED,&t);
     while(!stop){
          copy_device_data(&D.d);
          /* ignore write fails, we just drop the data */
          write(fd,&D,sizeof(D));
          timespec_add_ns(&t,DELAY_NS);
          clock_nanosleep(CLOCK_RTL_SCHED,
                          TIMER_ABSTIME,
                          &t,
                          NULL);
     }
     return (void *)&stop;
}

This code is placed inside an ordinary Linux kernel module which has
the format:

INCLUDES
GLOBAL VARIABLES AND DEFINITIONS
INITIALIZE CODE
CLEANUP CODE
REALTIME THREAD CODE

The init_module code for our example will create "fifo" to send data to
the user process and then create the thread. The cleanup_module asks
the thread to shut down, waits for it, and then closes the fifo.

pthread_t T;
int fd;
int stop = 0;
#define DELAY_NS 500000 // 500 microseconds

int init_module(void){
     if ((fd=open("/dev/rtf0",
                   O_WRONLY |
                   O_NONBLOCK |
                   O_CREAT ))
         < 0)
     {
          rtl_printf("Example cannot open fifo\n");
          rtl_printf("Error number is %d\n",errno);
          return -1;
     }
     if( pthread_create(&T,NULL,my_code,NULL))
     {
          close(fd);
          rtl_printf("Cannot create thread\n");
          return -1;
     }
     return 0;
}

void cleanup_module(void){
     stop = 1;
     pthread_join(T,NULL);
     close(fd);
}

If you just wanted to log the data produced by this code, the Linux
shell command cat < /dev/rtf0 > logfile would do the trick. A two line
awk script could convert the data into a format where it could be
displayed by gplot. Alternatively, a simply C program could take the
data and do some analysis, or a Perl script could send the data over a
network. By adding a second fifo, we could have the user interface
control the period or other properties of the sampling program. The
RTLinux front end RTiC Lab automates much of this.

Full working code can be found at http://www.rtlinux.com/
RTLinuxExamples .

Discussion

On a standard x86 PC RTLinux interrupt latencies are 20 microseconds,
worst case. On hardware with better timers and interrupt handlers we do
much better. For example, the AMD Elan520 a 133MHz processor shows
worst case interrupt latencies of under 7microseconds and on the
Motorola M8260 worst case seems to be no more than 4 microseconds. Of
course, PC architecture is not created with realtime in mind and badly
behaved hardware will cause long delays. For example, some low-end
video cards will hold the PCI bus for millisecond intervals if their
internal fifos are filled! But, with sensible hardware choices, RTLinux
pushes the boundaries of interrupt response down to a level where
commodity microprocessors can replace special purpose devices.

One design principle that 6 years of development has validated is our
principle that anything that can go into the general purpose operating
system should go into the general purpose operating system. Realtime
applications should be ruthlessly separated into the part that must be
realtime and the part that does not have hard time constraints. This is
a design constraint that conflicts with the programming model in IRIX,
VxWorks, and other traditional ``realtime" operating systems and it
requires an adjustment on the part of the programmer. But our decoupled
design is at the heart of why RTLinux is reliable and fast. We require
programmers to identify what sections of their code is realtime and our
operating system takes advantage of that information. In practice, with
tens of thousands of applications, this seems to be a natural way to
design modular applications. There are some difficulties porting legacy
code, but the user mode RTLinux signal capability reduces this
difficulty. In fact, ordinary Linux, provides better realtime
performance than many ``realtime" operating systems and we have found
that turning a non-Linux realtime application into a multithreaded
Linux application with RTLinux signal handlers is reasonably simple.

Since RTLinux was introduced, a number of ``real time" Linuxes have
been developed and marketed and advocates for those approaches often
claim to be escaping from the rigors our our decoupled model. Lineo
provides a system called RTAI which started as a variant of RTLinux and
has since grown considerably. MontaVista is attempting to introduce
some techniques from the IRIX Operating System into Linux with a
``fully preemptable kernel". There is, however, no reason to suppose
that re-creating VxWorks in Linux (Lineo) or re-creating IRIX in Linux
(MontaVista) will produce improvements on the originals. The
open-source development model, however, means that good work will enter
the mainstreams of code development. There are several companies and
researchers working on making Linux itself lower latency and this will
extend the range of RTLinux applications where a soft realtime Linux
component can be connected to hard realtime RTLinux threads. And RTAI,
whose main developer is an independent University researcher, is free
to experiment with APIs. The RTLinux user mode realtime signals were
developed after a far more ambitious mechanism in RTAI proved to be
useful. It's more important for us to retain a POSIX compatible and
small API, so RTLinux is much more conservative about adding new
features and much more likely to push new functionality into the
existing framework.

As an example of how new functionality can be accomodated within the
POSIX framework consider RTLinux/SMP. POSIX does not substantially
address the issue of multiple processors. In RTLinux, each processor of
a SMP system runs its own copy of the RTLinux realtime process (sharing
minimal data). Allocation of threads and signal handlers to processors
is completeley determined by the application. RTLinux adds a processor
identifier to the POSIX thread attribute and when a thread is created
the application can assign it to a processor. Efficient cross-processor
control is a difficult problem and we were reluctant to introduce
hidden synchronzation points or operations that would cause significant
cache disruption -- e.g by migrating tasks between processors. By
sending a suspend signal to the general purpose OS thread on a
particular processor, it is possible to dedicate that processor to
RTLinux. When the Linux thread on the target processor receives the
signal, it will run an idle loop. As a result, the realtime
applications on that processor can generally run within the processor
cache: greatly reducing latency. All this is done with no additions to
the POSIX threads API.

About this document ...

Copyright: 2001 FSM Labs, Inc.
-----------------------------------------------------------------------

Footnotes

... Yodaiken1
    yodaiken@fsmlabs.com
... beautiful2
    A word not usually associated with POSIX

-----------------------------------------------------------------------
[ ] [] [ ]

Michael Barabanov 2001-07-26
