[etherlab-users] MultiTask/Process Application

[Tom Wong-Cornall]

On Sat, Oct 19, 2019 at 02:58:06PM NZDT, Peter Dupej (student) <p.dupej.1 at research.gla.ac.uk> wrote:

>Is it possible with EtherLAB to write a RT Application with multiple tasks 
>running on multiple cores accessing a single master device similarly as 
>modern TwinCAT 3 allows?

Yes, with caveats; I imagine this is how most of us are using EtherLab (at 
least those who use it in userspace).

>I understand that there can only be one master process reserving the network 
>device, but would you be able to configure a set of 2-3 additional processes 
>that would communicate with the main one to distribute the computation load 
>onto multiple cores?

That would be a reasonably standard way to do it, either separate processes or 
separate threads. We use it like this, and I would guess many others would 
find similarities in the following (although the RTAI folks would have a few 
quirks):

 - Application runs in userspace, communicating with EtherLab running in 
   kernelspace (PREEMPT-RT in our case)

 - One thread gets given highest scheduling priority and dedicated CPU; this 
   thread alone talks with EtherLab; although it does do some calculations 
   (like generating servo trajectories etc.), mostly it's fairly dumb and just 
   copies data to/fro EtherLab

 - Other threads in same process do some heavier calculations and 
   communication with outside devices (CANbus, old-fashioned Modbus/TCP, RS232 
   etc.), and communicate with this thread via standard process memory 
   protected by normal pthread mutexes

 - These other threads run on separate cores, still with high priority (but 
   not as high as the EtherCAT thread). We use things like "try-lock" on the 
   mutexes to try and avoid priority inversion, as our EtherCAT setup uses 
   distributed clocks and is very picky about latency.
 
 - These other threads also communicate with a number of other processes on 
   the same box via all sorts of IPC mechanisms like shared memory, named 
   pipes, local sockets etc. These other processes do things like show GUIs, 
   log to disk, load/save production files for the machine to process etc.

If you're using a standard kernel (or even one with PREEMPT-RT patch), and 
running in userspace, then your `EtherCAT process' is really just an ordinary 
process with all the normal capabilities built into it for communicating with 
other processes or threads.

The only thing to be aware of is that many EtherCAT applications are quite 
sensitive of scheduling latency, especially if you're running distributed 
clocks, so the natural bias is to use fast low-latency IPC mechanisms that 
won't cause your EtherCAT process to get `unscheduled'. Shared memory is good 
for this, maybe serialising/deserialising XML down a local TCP/IP socket would 
not be so wise...

Cheers,

Tom