[etherlab-dev] Possible Realtime Issues with Ethercat Master and RT Preempt Kernel

Mon Feb 1 14:45:01 CET 2016

Hi Matthias

The e1000e Tx/Rx IRQ threads must have a priority that is higher than
the priority of the realtime thread (the one that calls
ecrt_send/receive) and all throttling done by the e1000e must be
disabled. The priority of the Ethercat-OP thread is not really a problem.

But, we had a problem on some e1000e revisions when a network statistic
watch was running (e.g. gnome-system-monitor)
     > watch -n 0.1 cat /sys/class/net/eth1/statistics/*

Use the kernel tracers:
Start trace and after a few seconds, hit Ctrl-C:
     > sudo trace-cmd record -e sched -e 'irq_handler_*' -p function -l
'ec_*' -l 'ecrt_*' -l 'e1000*' -b 16384 -s 250 -r 90
Watch trace (use thread filters and zoom)
     > kernelshark

BTW. our machines use a 250microsecond cycle time!

Regards
Martin

On 01.02.2016 14:01, Dr.-Ing. Matthias Schöpfer wrote:
> Hi Armin,
>
> thanks for your message. We are using this kernel version in some other
> realtime setups successfully. cyclictest and hwlatdetect show, that the
> system is running as one would expect.
>
> In these setups, we usually got a hardware interrupt, and it was usually
> necessary to boost the priority of the threaded interrupt to be totally
> safe even in situations of high (I/O) load.
>
> So, my question is, if it could be a fruitful endeavor to look into the
> etherlab kernel modules to set the priority for ethercat related
> tasks/threads. From looking from outside, I see a EtherCAT-OP thread,
> which I can easily put to RT prio (and did), but it seems that most work
> is done in the ksoftirqd, which as far as I know is of general use.
>
> I have not looked into the sources, and I am not 100% sure if this is
> the source of some problems we see here. Therefore my question, has
> anyone looked into it? If it is just about dispatching the work to a
> dedicated kernel thread instead of ksoftirqd the required changes would
> possibly small...
>
> Thanks and Regards,
>
>       Mattthias
>
> On 02/01/2016 11:14 AM, Armin Steinhoff wrote:
>> Hi Dr. Schöpfer,
>>
>> please have a look to the test rack at the OASDL homepage and you will
>> see that the same release of PREEMPT_RT is performing very different on
>> different motherboards.
>>
>> IMHO ... the most important issue is the SMI interrupt and the triggered
>> execution of the low level firmware of the motherboard.
>> Also a transaction on the PCI bus using DMA transfers can increase the
>> response time on a single core ...
>>
>> Best Regards
>>
>> Armin
>>
>> http://www.steinhoff-automation.com
>>
>>
>> Dr.-Ing. Matthias Schöpfer schrieb:
>>> Hi Thomas Winding,
>>>
>>> thanks for your answer!
>>>
>>> We are running kernel 3.12.31-rt45, since the newer kernel versions do
>>> not seem to run as reliable regarding real time at least on our hardware.
>>>
>>> ethercat is from the mecurical repository and we use the e1000e driver.
>>>
>>> Most of our problems we might have solved, since we had an issue with
>>> our cycle time / calculations where sometimes, our cycle lasted 600-800
>>> microseconds, which seems to be way to long. We are now down to < 220
>>> microseconds.
>>>
>>> In rare cases, when we start the software, we still constantly running
>>> in the mentioned issues. When we stop and restart, everything is fine. I
>>> wonder, if we sometimes hit a misfortuned timing. As of lately, I was
>>> not able to reproduce it.
>>>
>>> Regarding your suspicion: I wonder, if it is better to sync the
>>> transmits to the 1 ms cycle instead of the receives?!
>>>
>>> Regards,
>>>
>>>      Matthias SchÃ¶pfer
>>>
>>> On 02/01/2016 09:07 AM, Thomas Winding wrote:
>>>> Hi Matthias SchÃ¶pfer
>>>>
>>>> I have seen the same problem. I am also running at 1 ms and using
>>>> clock_nanosleep.
>>>>
>>>>    I suspect that the problem arise when you send a new telegram
>>>> before you have received the previously send telegram.
>>>>
>>>> Which version of the ethercat are you using?
>>>> Which version of kernel are you using?
>>>>
>>>> Best regard,
>>>>
>>>> Thomas Winding
>>>>
>>>> -----Original Message-----
>>>> From: etherlab-dev [mailto:etherlab-dev-bounces at etherlab.org] On
>>>> Behalf Of Dr.-Ing. Matthias SchÃ¶pfer
>>>> Sent: 26. januar 2016 14:22
>>>> To: etherlab-dev at etherlab.org
>>>> Subject: [etherlab-dev] Possible Realtime Issues with Ethercat Master
>>>> and RT Preempt Kernel
>>>>
>>>> Hi!
>>>>
>>>> We started using etherlab/ethercat and are quite impressed. Nice work!
>>>>
>>>> We are running Linux with a RT_PREEMPT Kernel and e1000e ethercat
>>>> driver. We have to run at a cycle time of 1ms and we have jitter from
>>>> clock_nanosleep of about 15 microsecs max.
>>>>
>>>> Nevertheless, we suffer from time to time from these:
>>>>
>>>> EtherCAT WARNING 0: 2 datagrams UNMATCHED!
>>>> EtherCAT 0: Domain 0: Working counter changed to 9/9.
>>>> EtherCAT 0: Domain 0: Working counter changed to 0/9.
>>>>
>>>> Especially, when we apply load to the system. From previous projects,
>>>> I experienced these effects when IRQ/Kernel Thread was not set to
>>>> appropriate RT Level.
>>>>
>>>> My Question: has anybody experienced similar problems, and would it
>>>> be worth to investigate it. And if I decide to patch the kernel
>>>> module, where is a good starting point.
>>>>
>>>> Thanks and regards,
>>>>
>>>>      Matthias SchÃ¶pfer
>>>>
>>>> --
>>>> Dr. Matthias SchÃ¶pfer
>>>> mz robolab GmbH
>>>> Marie-Curie-Str. 1
>>>> 53359 Rheinbach
>>>>
>>>> Office: +49 2226 83600 00
>>>> Fax: +49 2226 83600 11
>>>> Email: schoepfer at robolab.de
>>>>
>>>> mz robolab GmbH
>>>> Vertretungsberechtigte GeschÃ€ftsfÃŒhrer: Dr. RÃŒdiger MaaÃŸ, Ralf
>>>> Schulte Registergericht Amtsgericht Bonn Registernummer HRB 10595
>>>> ---
>>>> This e-mail may contain confidential and/or privileged information.
>>>> If you are not the intended recipient (or have received this e-mail in
>>>> error) please notify the sender immediately and destroy this e-mail.
>>>> Any unauthorised copying, disclosure or distribution of the material
>>>> in this e-mail is strictly forbidden.
>>>>
>>>> Diese E-Mail enthÃ€lt vertrauliche und/oder rechtlich geschÃŒtzte
>>>> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese
>>>> E-Mail irrtÃŒmlich erhalten haben, informieren Sie bitte sofort den
>>>> Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie
>>>> die unbefugte Weitergabe dieser Mail ist nicht gestattet.
>>>> ---
>>>> P    please consider the environment before printing this e-mail
>>>> _______________________________________________
>>>> etherlab-dev mailing list
>>>> etherlab-dev at etherlab.org
>>>> http://lists.etherlab.org/mailman/listinfo/etherlab-dev
>>>>

Note:

This e-mail is for the named person's use only. It may contain confidential and/or privileged information. If you have received this e-mail in error, please notify the sender immediately and delete the material from any system. Any unauthorized copying, disclosure, distribution or other use of this information by persons or entities other than the intended recipient is prohibited.

Thank You.