[etherlab-dev] Possible Realtime Issues with Ethercat Master and RT Preempt Kernel

Graeme Foot Graeme.Foot at touchcut.com
Wed Feb 3 23:18:54 CET 2016 

Hi,

I saw some patches sent to the forum a while ago with regards to adding a call to check if the return packets had been received yet.  Not sure if it was a blocking call or one that you had to poll.  Searching for "Knowing when the packet has finished cycle" and "Waiting for network receive".

One of the other process cycle options is to split read PDO's and write PDO's into two different domains.  Some of the recent Beckhoff TwinCAT literature describes how this can be useful in reducing response times.

Graeme.

-----Original Message-----
From: Tillman, Scott [mailto:Scott.Tillman at bhemail.com] 
Sent: Wednesday, 3 February 2016 9:02 p.m.
To: Graeme Foot; Dr.-Ing. Matthias Schöpfer; etherlab-dev at etherlab.org
Subject: RE: [etherlab-dev] Possible Realtime Issues with Ethercat Master and RT Preempt Kernel

Hi Graeme,

Since you brought up the typical process cycle: I have been using a process similar the second one you describe.  I was very surprised when I was doing my initial development that the output frame and the return frame were overlaid, requiring double buffering of the output data.  It seems like you should be able to configure the domain to place the return data in a separate (possibly neighboring) memory area.  As it is the double buffering is the same idea, but causes an extra memcpy just prior to sending the domain data.

More problematic is the absence of any way to block (in user-space) waiting for the domain's return packet.  As it is I am setting up my clock at 0.5ms to handle a 1ms frame time:

 - sleep to output time for frame N
 - memcpy buffered output states to domain frame
 - ecrt_domain_queue and ecrt_master_send
 - pre-calculate data for next output cycle (like servo trajectories which don't need immediate input data)
 - sleep to input time for frame N return data
 - trigger computations requiring latest input data (servo loop closure for instance)
 - notify lower priority tasks of input changes (sensor reaction logics)
 - sleep to output time for frame N+1

This is working well and has a near constant frame transmit delay minimizing output frame jitter.  It would be nice to have some kind of blocking system where I could just setup a thread to block waiting for the next return frame rather than assuming a return time of, at most, half out actual cycle time.

Are these two things there somewhere and I've just missed them, or is there a good reason they haven't been implemented?  It seems like these two items would minimize the overhead and maximize the processing time available for most applications.

-Scott Tillman

-----Original Message-----
From: etherlab-dev [mailto:etherlab-dev-bounces at etherlab.org] On Behalf Of Graeme Foot
Sent: Tuesday, February 02, 2016 6:36 PM
To: Dr.-Ing. Matthias Schöpfer <schoepfer at robolab.de>; etherlab-dev at etherlab.org
Subject: Re: [etherlab-dev] Possible Realtime Issues with Ethercat Master and RT Preempt Kernel

Hi,

Just a thought for the day, how many EtherCAT slaves do you have and what's the last slaves "DC system time transmission delay"?

Use the command "ethercat slaves -v" and check the last slaves "DC system time transmission delay" value.  If you have a linear topology then the complete wire transmission time (master -> slaves -> master) will be approximately twice this value.  Plus some overhead for the network card driver to process the sending and receiving.

If you use the traditional cycle of:
- cycle sleep
- master_receive
- domain_process
- calc
- domain_queue
- master_send
- cycle sleep
- ...

Then you only have the sleep time available for the EtherCAT frames to go out and return.  If the calc time takes too long on a particular cycle then there may not be enough time for the frames to return and be ready.


Another problem is if you do have a widely varying calc time then you get a widely varying send time (and slave read time).  If using distributed clocks this may not be a problem (unless you miss the DC window), but without DC synchronization you can get some variance with when the IO on the slaves are being switched (or when a motor gets a new position Target).

The way I get around this is to use the following sequence:
- cycle sleep
- master_receive
- domain_process
- write cached values to domains
- domain_queue
- master_send
- calc - writing to cached values
- cycle sleep
- ...

The drawbacks of the above is that you need to calc and write values to a cached location else the master_receive / domain_process calls will overwrite your calced values with the returned packets data.  It also adds an extra cycle before read data is available.  You also have extra time overhead in managing the cached values and writing them before the send.  The upside is you have a consistent send time and have nearly all of the scan period available for the frames to be sent and received allowing you to perform the calculations in parallel.  Due to this you can also potentially reduce the overall cycle time, negating the problem of the reads being delayed an extra cycle.


Regards,
Graeme.


-----Original Message-----
From: etherlab-dev [mailto:etherlab-dev-bounces at etherlab.org] On Behalf Of Dr.-Ing. Matthias Schöpfer
Sent: Monday, 1 February 2016 9:27 p.m.
To: etherlab-dev at etherlab.org
Subject: Re: [etherlab-dev] Possible Realtime Issues with Ethercat Master and RT Preempt Kernel

Hi Thomas Winding,

thanks for your answer!

We are running kernel 3.12.31-rt45, since the newer kernel versions do not seem to run as reliable regarding real time at least on our hardware.

ethercat is from the mecurical repository and we use the e1000e driver.

Most of our problems we might have solved, since we had an issue with our cycle time / calculations where sometimes, our cycle lasted 600-800 microseconds, which seems to be way to long. We are now down to < 220 microseconds.

In rare cases, when we start the software, we still constantly running in the mentioned issues. When we stop and restart, everything is fine. I wonder, if we sometimes hit a misfortuned timing. As of lately, I was not able to reproduce it.

Regarding your suspicion: I wonder, if it is better to sync the transmits to the 1 ms cycle instead of the receives?!

Regards,

	Matthias Schöpfer

On 02/01/2016 09:07 AM, Thomas Winding wrote:
> Hi Matthias Schöpfer
> 
> I have seen the same problem. I am also running at 1 ms and using clock_nanosleep.
> 
>  I suspect that the problem arise when you send a new telegram before you have received the previously send telegram.
> 
> Which version of the ethercat are you using?
> Which version of kernel are you using?
> 
> Best regard,
> 
> Thomas Winding
> 
> -----Original Message-----
> From: etherlab-dev [mailto:etherlab-dev-bounces at etherlab.org] On 
> Behalf Of Dr.-Ing. Matthias Schöpfer
> Sent: 26. januar 2016 14:22
> To: etherlab-dev at etherlab.org
> Subject: [etherlab-dev] Possible Realtime Issues with Ethercat Master 
> and RT Preempt Kernel
> 
> Hi!
> 
> We started using etherlab/ethercat and are quite impressed. Nice work!
> 
> We are running Linux with a RT_PREEMPT Kernel and e1000e ethercat driver. We have to run at a cycle time of 1ms and we have jitter from clock_nanosleep of about 15 microsecs max.
> 
> Nevertheless, we suffer from time to time from these:
> 
> EtherCAT WARNING 0: 2 datagrams UNMATCHED!
> EtherCAT 0: Domain 0: Working counter changed to 9/9.
> EtherCAT 0: Domain 0: Working counter changed to 0/9.
> 
> Especially, when we apply load to the system. From previous projects, I experienced these effects when IRQ/Kernel Thread was not set to appropriate RT Level.
> 
> My Question: has anybody experienced similar problems, and would it be worth to investigate it. And if I decide to patch the kernel module, where is a good starting point.
> 
> Thanks and regards,
> 
> 	Matthias Schöpfer
> 
> --
> Dr. Matthias Schöpfer
> mz robolab GmbH
> Marie-Curie-Str. 1
> 53359 Rheinbach
> 
> Office: +49 2226 83600 00
> Fax: +49 2226 83600 11
> Email: schoepfer at robolab.de
> 
> mz robolab GmbH
> Vertretungsberechtigte Geschäftsführer: Dr. Rüdiger Maaß, Ralf Schulte 
> Registergericht Amtsgericht Bonn Registernummer HRB 10595
> ---
> This e-mail may contain confidential and/or privileged information. If 
> you are not the intended recipient (or have received this e-mail in
> error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
> 
> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.
> ---
> P    please consider the environment before printing this e-mail
> _______________________________________________
> etherlab-dev mailing list
> etherlab-dev at etherlab.org
> http://lists.etherlab.org/mailman/listinfo/etherlab-dev
> 

--
Dr. Matthias Schöpfer
mz robolab GmbH
Marie-Curie-Str. 1
53359 Rheinbach

Office: +49 2226 83600 00
Fax: +49 2226 83600 11
Email: schoepfer at robolab.de

mz robolab GmbH
Vertretungsberechtigte Geschäftsführer: Dr. Rüdiger Maaß, Ralf Schulte Registergericht Amtsgericht Bonn Registernummer HRB 10595
---
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in
error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.
---
P    please consider the environment before printing this e-mail
_______________________________________________
etherlab-dev mailing list
etherlab-dev at etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-dev
_______________________________________________
etherlab-dev mailing list
etherlab-dev at etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-dev

More information about the etherlab-dev mailing list