[etherlab-dev] Possible Realtime Issues with Ethercat Master and RT Preempt Kernel

[Graeme Foot]

Hi,

Just a thought for the day, how many EtherCAT slaves do you have and what's the last slaves "DC system time transmission delay"?

Use the command "ethercat slaves -v" and check the last slaves "DC system time transmission delay" value.  If you have a linear topology then the complete wire transmission time (master -> slaves -> master) will be approximately twice this value.  Plus some overhead for the network card driver to process the sending and receiving.

If you use the traditional cycle of:
- cycle sleep
- master_receive
- domain_process
- calc
- domain_queue
- master_send
- cycle sleep
- ...

Then you only have the sleep time available for the EtherCAT frames to go out and return.  If the calc time takes too long on a particular cycle then there may not be enough time for the frames to return and be ready.


Another problem is if you do have a widely varying calc time then you get a widely varying send time (and slave read time).  If using distributed clocks this may not be a problem (unless you miss the DC window), but without DC synchronization you can get some variance with when the IO on the slaves are being switched (or when a motor gets a new position Target).

The way I get around this is to use the following sequence:
- cycle sleep
- master_receive
- domain_process
- write cached values to domains
- domain_queue
- master_send
- calc - writing to cached values
- cycle sleep
- ...

The drawbacks of the above is that you need to calc and write values to a cached location else the master_receive / domain_process calls will overwrite your calced values with the returned packets data.  It also adds an extra cycle before read data is available.  You also have extra time overhead in managing the cached values and writing them before the send.  The upside is you have a consistent send time and have nearly all of the scan period available for the frames to be sent and received allowing you to perform the calculations in parallel.  Due to this you can also potentially reduce the overall cycle time, negating the problem of the reads being delayed an extra cycle.


Regards,
Graeme.


-----Original Message-----
From: etherlab-dev [mailto:etherlab-dev-bounces at etherlab.org] On Behalf Of Dr.-Ing. Matthias Schöpfer
Sent: Monday, 1 February 2016 9:27 p.m.
To: etherlab-dev at etherlab.org
Subject: Re: [etherlab-dev] Possible Realtime Issues with Ethercat Master and RT Preempt Kernel

Hi Thomas Winding,

thanks for your answer!

We are running kernel 3.12.31-rt45, since the newer kernel versions do not seem to run as reliable regarding real time at least on our hardware.

ethercat is from the mecurical repository and we use the e1000e driver.

Most of our problems we might have solved, since we had an issue with our cycle time / calculations where sometimes, our cycle lasted 600-800 microseconds, which seems to be way to long. We are now down to < 220 microseconds.

In rare cases, when we start the software, we still constantly running in the mentioned issues. When we stop and restart, everything is fine. I wonder, if we sometimes hit a misfortuned timing. As of lately, I was not able to reproduce it.

Regarding your suspicion: I wonder, if it is better to sync the transmits to the 1 ms cycle instead of the receives?!

Regards,

	Matthias Schöpfer

On 02/01/2016 09:07 AM, Thomas Winding wrote:
> Hi Matthias Schöpfer
> 
> I have seen the same problem. I am also running at 1 ms and using clock_nanosleep.
> 
>  I suspect that the problem arise when you send a new telegram before you have received the previously send telegram.
> 
> Which version of the ethercat are you using?
> Which version of kernel are you using?
> 
> Best regard,
> 
> Thomas Winding
> 
> -----Original Message-----
> From: etherlab-dev [mailto:etherlab-dev-bounces at etherlab.org] On 
> Behalf Of Dr.-Ing. Matthias Schöpfer
> Sent: 26. januar 2016 14:22
> To: etherlab-dev at etherlab.org
> Subject: [etherlab-dev] Possible Realtime Issues with Ethercat Master 
> and RT Preempt Kernel
> 
> Hi!
> 
> We started using etherlab/ethercat and are quite impressed. Nice work!
> 
> We are running Linux with a RT_PREEMPT Kernel and e1000e ethercat driver. We have to run at a cycle time of 1ms and we have jitter from clock_nanosleep of about 15 microsecs max.
> 
> Nevertheless, we suffer from time to time from these:
> 
> EtherCAT WARNING 0: 2 datagrams UNMATCHED!
> EtherCAT 0: Domain 0: Working counter changed to 9/9.
> EtherCAT 0: Domain 0: Working counter changed to 0/9.
> 
> Especially, when we apply load to the system. From previous projects, I experienced these effects when IRQ/Kernel Thread was not set to appropriate RT Level.
> 
> My Question: has anybody experienced similar problems, and would it be worth to investigate it. And if I decide to patch the kernel module, where is a good starting point.
> 
> Thanks and regards,
> 
> 	Matthias Schöpfer
> 
> --
> Dr. Matthias Schöpfer
> mz robolab GmbH
> Marie-Curie-Str. 1
> 53359 Rheinbach
> 
> Office: +49 2226 83600 00
> Fax: +49 2226 83600 11
> Email: schoepfer at robolab.de
> 
> mz robolab GmbH
> Vertretungsberechtigte Geschäftsführer: Dr. Rüdiger Maaß, Ralf Schulte 
> Registergericht Amtsgericht Bonn Registernummer HRB 10595
> ---
> This e-mail may contain confidential and/or privileged information. If 
> you are not the intended recipient (or have received this e-mail in
> error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
> 
> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.
> ---
> P    please consider the environment before printing this e-mail
> _______________________________________________
> etherlab-dev mailing list
> etherlab-dev at etherlab.org
> http://lists.etherlab.org/mailman/listinfo/etherlab-dev
> 

--
Dr. Matthias Schöpfer
mz robolab GmbH
Marie-Curie-Str. 1
53359 Rheinbach

Office: +49 2226 83600 00
Fax: +49 2226 83600 11
Email: schoepfer at robolab.de

mz robolab GmbH
Vertretungsberechtigte Geschäftsführer: Dr. Rüdiger Maaß, Ralf Schulte Registergericht Amtsgericht Bonn Registernummer HRB 10595
---
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in
error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.
---
P    please consider the environment before printing this e-mail
_______________________________________________
etherlab-dev mailing list
etherlab-dev at etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-dev