[etherlab-dev] Possible Realtime Issues with Ethercat Master and RT Preempt Kernel

Mon Feb 1 14:01:16 CET 2016

Hi Armin,

thanks for your message. We are using this kernel version in some other
realtime setups successfully. cyclictest and hwlatdetect show, that the
system is running as one would expect.

In these setups, we usually got a hardware interrupt, and it was usually
necessary to boost the priority of the threaded interrupt to be totally
safe even in situations of high (I/O) load.

So, my question is, if it could be a fruitful endeavor to look into the
etherlab kernel modules to set the priority for ethercat related
tasks/threads. From looking from outside, I see a EtherCAT-OP thread,
which I can easily put to RT prio (and did), but it seems that most work
is done in the ksoftirqd, which as far as I know is of general use.

I have not looked into the sources, and I am not 100% sure if this is
the source of some problems we see here. Therefore my question, has
anyone looked into it? If it is just about dispatching the work to a
dedicated kernel thread instead of ksoftirqd the required changes would
possibly small...

Thanks and Regards,

	Mattthias

On 02/01/2016 11:14 AM, Armin Steinhoff wrote:
> 
> Hi Dr. Schöpfer,
> 
> please have a look to the test rack at the OASDL homepage and you will
> see that the same release of PREEMPT_RT is performing very different on
> different motherboards.
> 
> IMHO ... the most important issue is the SMI interrupt and the triggered
> execution of the low level firmware of the motherboard.
> Also a transaction on the PCI bus using DMA transfers can increase the
> response time on a single core ...
> 
> Best Regards
> 
> Armin
> 
> http://www.steinhoff-automation.com
> 
> 
> Dr.-Ing. Matthias Schöpfer schrieb:
>> Hi Thomas Winding,
>>
>> thanks for your answer!
>>
>> We are running kernel 3.12.31-rt45, since the newer kernel versions do
>> not seem to run as reliable regarding real time at least on our hardware.
>>
>> ethercat is from the mecurical repository and we use the e1000e driver.
>>
>> Most of our problems we might have solved, since we had an issue with
>> our cycle time / calculations where sometimes, our cycle lasted 600-800
>> microseconds, which seems to be way to long. We are now down to < 220
>> microseconds.
>>
>> In rare cases, when we start the software, we still constantly running
>> in the mentioned issues. When we stop and restart, everything is fine. I
>> wonder, if we sometimes hit a misfortuned timing. As of lately, I was
>> not able to reproduce it.
>>
>> Regarding your suspicion: I wonder, if it is better to sync the
>> transmits to the 1 ms cycle instead of the receives?!
>>
>> Regards,
>>
>>     Matthias SchÃ¶pfer
>>
>> On 02/01/2016 09:07 AM, Thomas Winding wrote:
>>> Hi Matthias SchÃ¶pfer
>>>
>>> I have seen the same problem. I am also running at 1 ms and using
>>> clock_nanosleep.
>>>
>>>   I suspect that the problem arise when you send a new telegram
>>> before you have received the previously send telegram.
>>>
>>> Which version of the ethercat are you using?
>>> Which version of kernel are you using?
>>>
>>> Best regard,
>>>
>>> Thomas Winding
>>>
>>> -----Original Message-----
>>> From: etherlab-dev [mailto:etherlab-dev-bounces at etherlab.org] On
>>> Behalf Of Dr.-Ing. Matthias SchÃ¶pfer
>>> Sent: 26. januar 2016 14:22
>>> To: etherlab-dev at etherlab.org
>>> Subject: [etherlab-dev] Possible Realtime Issues with Ethercat Master
>>> and RT Preempt Kernel
>>>
>>> Hi!
>>>
>>> We started using etherlab/ethercat and are quite impressed. Nice work!
>>>
>>> We are running Linux with a RT_PREEMPT Kernel and e1000e ethercat
>>> driver. We have to run at a cycle time of 1ms and we have jitter from
>>> clock_nanosleep of about 15 microsecs max.
>>>
>>> Nevertheless, we suffer from time to time from these:
>>>
>>> EtherCAT WARNING 0: 2 datagrams UNMATCHED!
>>> EtherCAT 0: Domain 0: Working counter changed to 9/9.
>>> EtherCAT 0: Domain 0: Working counter changed to 0/9.
>>>
>>> Especially, when we apply load to the system. From previous projects,
>>> I experienced these effects when IRQ/Kernel Thread was not set to
>>> appropriate RT Level.
>>>
>>> My Question: has anybody experienced similar problems, and would it
>>> be worth to investigate it. And if I decide to patch the kernel
>>> module, where is a good starting point.
>>>
>>> Thanks and regards,
>>>
>>>     Matthias SchÃ¶pfer
>>>
>>> -- 
>>> Dr. Matthias SchÃ¶pfer
>>> mz robolab GmbH
>>> Marie-Curie-Str. 1
>>> 53359 Rheinbach
>>>
>>> Office: +49 2226 83600 00
>>> Fax: +49 2226 83600 11
>>> Email: schoepfer at robolab.de
>>>
>>> mz robolab GmbH
>>> Vertretungsberechtigte GeschÃ€ftsfÃŒhrer: Dr. RÃŒdiger MaaÃŸ, Ralf
>>> Schulte Registergericht Amtsgericht Bonn Registernummer HRB 10595
>>> ---
>>> This e-mail may contain confidential and/or privileged information.
>>> If you are not the intended recipient (or have received this e-mail in
>>> error) please notify the sender immediately and destroy this e-mail.
>>> Any unauthorised copying, disclosure or distribution of the material
>>> in this e-mail is strictly forbidden.
>>>
>>> Diese E-Mail enthÃ€lt vertrauliche und/oder rechtlich geschÃŒtzte
>>> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese
>>> E-Mail irrtÃŒmlich erhalten haben, informieren Sie bitte sofort den
>>> Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie
>>> die unbefugte Weitergabe dieser Mail ist nicht gestattet.
>>> ---
>>> P    please consider the environment before printing this e-mail
>>> _______________________________________________
>>> etherlab-dev mailing list
>>> etherlab-dev at etherlab.org
>>> http://lists.etherlab.org/mailman/listinfo/etherlab-dev
>>>
> 

-- 
Dr. Matthias Schöpfer
mz robolab GmbH
Marie-Curie-Str. 1
53359 Rheinbach

Office: +49 2226 83600 00
Fax: +49 2226 83600 11
Email: schoepfer at robolab.de

mz robolab GmbH
Vertretungsberechtigte Geschäftsführer: Dr. Rüdiger Maaß, Ralf Schulte
Registergericht Amtsgericht Bonn
Registernummer HRB 10595
---
This e-mail may contain confidential and/or privileged information. If
you are not the intended recipient (or have received this e-mail in
error) please notify the sender immediately and destroy this e-mail. Any
unauthorised copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese
E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den
Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie
die unbefugte Weitergabe dieser Mail ist nicht gestattet.
---
P    please consider the environment before printing this e-mail