Author: Tobit Flatscher (August 2021 - September 2022)
This is guide explains how one can develop inside a Docker container while being able to run real-time capable code on a Linux operating system. As such it walks you through:
- Basics of real-time systems and an overview of different real-time Linux approaches
- Set-up of a real-time system, in particular the installation of
PREEMPT_RT
supplying simple scripts for automatically re-compiling the kernel - Possible optimizations of a real-time system in order to minimise the latency
- Introduction into development with Docker as well as Docker-Compose and how you can set it up in Visual Studio Code, including a guide on how to use graphic user interfaces with Docker and tips on how to structure a ROS workspace with it.
- Required settings for a real-time capable container with a
PREEMPT_RT
patched host system - Benchmarking the real-time performance by means of
cyclictest
This can be useful for several different applications, in particular:
- Controlling real-time robotics hardware from Linux systems, e.g. over EtherCAT (by using EtherCAT masters such as SOEM or IgH) or setting up containers for other robotic components such as the Franka Emika Panda collaborative robotic arm
- Numerical control by turning your desktop or single-board computer into a SoftPLC e.g. with CodeSYS Control or using LinuxCNC
- Real-time programming in audio development such as with Juce
but may also apply to software developers developing any other form of real-time capable code.
There are different ways of turning a vanilla Linux system into a real-time capable operating system. As outlined in doc/realtime_basics/RealTimeLinux.md
PREEMPT_RT
is likely the most future-proof possibility as it is about to be included into the mainline of Linux.
The set-up of a real-time capable Docker with PREEMPT_RT
is quite straight forward. All you need is:
- A
PREEMPT_RT
-patched Linux host operating system - An arbitrary Docker container launched with the correct options so that it can set real-time priorities from inside the container as well as options for reducing the network latency
The manual set-up of PREEMPT_RT
takes quite a while (see doc/realtime_basics/PreemptRt.md
). You have two options, a custom re-compilation of the kernel or an installation from an existing Debian package.
The installation procedure either by compilation from source or from an existing Debian package is lined out in doc/realtime_basics/PreemptRt.md
. The same procedure can also be performed with the provided scripts src/install_debian_preemptrt
and src/compile_kernel_preemptrt
. src/install_debian_preemptrt
checks online if there are already pre-compiled PREEMPT_RT
packages available and lets you select a suiting version graphically, while src/compile_kernel_preemptrt
compiles the kernel from scratch and installs it. Before using the scripts be sure to make them executable on your system with $ sudo chmod +x install_debian_preemptrt
.
Start of by launching src/install_debian_preemptrt
and follow the installation instructions
$ cd src
$ ./install_debian_preemptrt
Afterwards you can reboot your system (be sure to select the correct kernel!) and should already be ready to go. You can check the kernel version with $ uname -r
to verify that you are using the correct kernel and the installation was indeed successful.
If the installation above fails (or for some good reason you have to compile the kernel yourself) you can use the src/compile_kernel_preemptrt
script.
You can launch it in two different ways:
$ cd src
$ ./compile_kernel_preemptrt
will install the required dependencies and then open a dialog which lets you browse the possible versions and available options manually, reducing the copy and paste operations.
If you supply a correct real-time patch version from the list of available ones as an input argument, launching the command with superuser privileges it will download all files, patch the kernel, create a Debian package if no official one is available and install it automatically.
$ cd src
$ sudo ./compile_kernel_preemptrt 5.10.78-rt55
This might be helpful for deploying a new kernel automatically on a remote system. The possible version numbers to be passed as arguments can be found at here.
After having patched your system and restarted it, booting into the freshly installed kernel (see doc/realtime_basics/ChangeBootOrder.md
), you should already be good to go to launch a real-time capable Docker. In case you do not intend to use root
as the user inside the Docker you furthermore will have to have give yourself a name of a user that belongs to a group with real-time privileges on your host computer. How this can be done can be found in doc/realtime_basics/PreemptRt.md
.
After having successfully installed PREEMPT_RT
, it is sufficient to execute the Docker with the options:
cap_add:
- SYS_NICE
ulimits:
rtprio: 99
rttime: -1 # corresponds to 'unlimited'
memlock: 8428281856
network_mode: host
Then any process from inside the Docker can set real-time priorities rtprio
(e.g. by calling ::pthread_setschedparam
from inside the C/C++ code or by using chrt
from the command line). Generally it is advisable to not set rtprio
to 99, refer to pull request #3 for more information.
This Github repository comes with a simple example that can be used to try it out. Inside the Docker container a cyclictest
is run to assess the real-time performance of the system. You can compare the outcome to running it on your local system. There should be virtually no difference between the two.
For launching the cyclictest
open the Docker by typing
$ docker-compose -f docker/docker-compose.yml up
then browse the folder benchmark/
and run the command
$ ./mklatencyplot.bash
This should create a latency histogram by measuring the difference between a thread's intended wake-up time and its actual wake up time. This measures any form of latency caused by hardware, firmware and operating system. For more information on this test refer to OSADL.