Developing for Multicore Systems

Approaches to Inter-OS Communication and Messaging for Multicore – Multi OS Systems

Realizing the potential of multicore processors involves more than simply running multiple RTOSs. Additional functions to manage communications control and messaging are needed and must be kept transparent to the operations.


  • Page 1 of 1
    Bookmark and Share

Article Media

The fact that embedded hypervisors can be used to run multiple operating systems independently in parallel and under real-time constraints on multicore platforms has become public knowledge. And quite a few of the market leaders have already consolidated their distributed old designs into cost-saving platforms using embedded virtualization. As operating systems don’t seem to be jacks of all trades, different requirements like those for real-time performance, strong graphics capabilities or efficient data processing and connectivity are typically solved by using different operating systems. Instead of still running these different operating systems each on separate hardware, developers now use embedded hypervisors to consolidate multiple platforms into a less expensive single-board solution with one or more multicore CPUs. Such a design allows the parallel execution of multiple operating systems without them influencing one another. Since the deployed operating systems don’t know of each other’s existence, the integration effort can be kept to a minimum.

Now imagine taking the first steps toward a modern, optimized system architecture: writing a concept and defining which operating systems should be used, which CPU(s) to be assigned to which OS, and specifying which devices should be assigned to the individual virtual machines for their exclusive use. Maybe the concept even includes details like the present cache topology of the available CPUs and an intelligent decision about which OSs should have their own, exclusive cache available to them and which OSs may share caches with others. In a user friendly, easy to configure hypervisor, like the RTS Hypervisor, the implementation is a walk in the park. Little changes made to the supplied example configuration file will already allow the system to start in the intended setup.

Once all the software components have been defined and all the hardware resources are partitioned, the details of hypervisors come into play: How will these cleanly separated, partitioned, hard and software modules work as a simple to use, integrated system that can be turned into a product for a user?

For quite some time most hypervisors have offered a virtual network, providing an easy means for communication as if the separate operating systems were running on separate hardware, connected with a physical network cable. Especially if porting an existing solution of multiple hardware boards to a modern design, this functionality is often used to reduce porting effort.

After these initial steps, unfortunately, only a few of the advantages of a multicore multi-OS design have been made use of: Power requirement (heat) and the bill of materials have been lowered, size and weight have been reduced, and the mean time between failure (MTBF) has been increased due to the reduced number of hardware components. The real advantages that would have required additional hardware on a distributed system can be obtained when digging deeper into the capabilities of some embedded hypervisors.

When running multiple operating systems on a single piece of hardware, a very simple but powerful method of communication now becomes available: shared memory. Two big advantages of shared memory over the virtual network are fast data exchange along with random access to data. To set up shared memory, only a name and the size of the memory need to be configured and assigned to the guest operating systems. Since the Intel multicore architecture provides cache coherence for RAM, this is also an attractive functionality to replace expensive DPRAM (Dual-Ported RAM) implementations. There are many good use cases for the use of shared memory. These include configuration settings that are generated at runtime, commonly used data bases, measurement data collected by one operating system but processed by another OS, or working with multiple operating systems on images that have been acquired by a camera (Figure 1).  

Figure 1
Wireless industrial M2M networks face challenges in terms of connectivity and security as well as the ability of the wireless gateway to recalculate and manage the network topology as the number of nodes continues to grow.

But shared memory is very useful for more than data exchange. In addition, the ability to pre-initialize a shared memory section with files at boot time can be very advantageous. In a common scenario, by running a traditional real-time operating system (RTOS) in parallel with Windows as a user interface, you may actually kill two birds with one stone. A file created by a user may, depending on the application at hand, contain project settings or even applications. This file can then be accessible by the RTOS via a pre-initialized shared memory section. If the hypervisor setting is such that the RTOS boots before a Windows boot, the RTOS may start its execution before Windows has even finished its boot. As a second advantage a user may, without changing the RTOS image or without even knowing about its existence, change the functionality of the RTOS simply by providing different content in the shared memory section.

An essential feature of most real-time operating systems is an event system. Programming a multicore multi-OS system with embedded virtualization can provide yet one more major advantage over old distributed hardware solutions. The RTS Hypervisor provides the ability to share events between operating systems, even if the installed operating systems are heterogeneous, e.g., Windows 7 running alongside an RTOS with a POSIX API like QNX Neutrino, Linux or VxWorks. These events may then be used just like semaphores for signaling resource usage, release of data in a shared memory section, to show that an application has started or even for fast data communication. In the RTS Hypervisor such an event system has been implemented using inter processor interrupts (IPI). In this implementation the hypervisor event system can be used for real-time tasks as well (Figure 2).

Figure 2
On the Intel Architecture, a modern Hypervisor can send IPIs (Inter Processor Interrupts) to and from any CPU to signal events in real time.

With these three capabilities, integrating distributed, multi hardware systems into a multicore multi-OS system, with applications now distributed over multiple virtual machines on one hypervisor platform, messaging and inter-OS communication becomes extremely powerful, fast and much more efficient.

Having powerful inter-OS communication and shared memory as well as an event system available, multicore multi-OS may even be the answer to some of the traditional symmetric multiprocessing  (SMP) problems. Even though SMP may be used on embedded virtualization, assigning application tasks or processes to separate, independent instances of an identical operating system and using the event system and shared memory to synchronize them instead, may result in much better performance than a traditional SMP solution. One of the advantages over regular SMP is that in this case, tasks and processes may now be pinned statically to a specific physical CPU. Even though real-time operating systems featuring SMP are not inherently non-deterministic, keeping an overview over the scheduling and interrupt handling is a challenge due to the fact that different tasks or processes may execute on different CPUs in parallel. Performance of multiple single core RTOSs in parallel can sometimes be much better than in an SMP solution,  because the synchronization overhead of symmetric multiprocessing may be higher than expected from an RTOS.

Given all of these benefits of a multicore multi-OS design, one could argue that one of the disadvantages of a virtualized platform could be that a user may only have access to the information about one virtual machine at a time, but not about the complete system. Information at runtime about which systems are currently running, how many CPUs are available, which devices are assigned to which guest operating system or details about the PCI bus topology, may not be accessible. In the RTS Hypervisor, for example, all this information provided by the hypervisor is readily accessible, not only in human readable format but also in XML, so this data can be processed in software. Using such an interface, user applications may generate new or modified hypervisor configurations automatically, or verify that all operating systems have been started in the originally intended configuration and on the genuine hardware board.

As multicore multi-OS systems have become more commonly used in the last few years, additional requirements have emerged. Uptime is important and safety and reliability are essential. With the above methods, some of these requirements can be satisfied with only limited effort.  However, on modern hypervisors, watchdogs are often available.  For example, each operating system running on the hypervisor is equipped with a watchdog counter incrementing constantly in a preset interval. These counters are accessible via an application programming interface (API), and if one of the guest operating systems stops counting, the appropriate action may be taken. Such an action could simply be an alarm, rebooting the operating system that failed, or any other measure the user wishes to take.

If designing and developing a system based on embedded virtualization, some attention should also be paid to the development process. During the development phase, an API for rebooting a single OS may save quite a bit of time. During development an OS may not respond to a reboot or shutdown request anymore, so it is also helpful if an API call may simply hard reset one of the virtual machines and then restart this guest OS, maybe even with a new operating system image.

Embedded hypervisors are very different from server-type virtual machine monitors. Guaranteeing hard real-time performance for an RTOS and providing access to the real, non-virtualized devices, are two of the key requirements an embedded hypervisor should provide. A number of real-time hypervisors have emerged over the last few years. Working with such a multicore multi-OS system is a different story. Simply running multiple operating systems in parallel is something most of them can do, but it is the details of inter-OS communication, guest OS control and messaging that will make the final product a success.  

Real-Time Systems
Ravensburg, Germany
+49 (0)751 359 558-0