[etherlab-users] Control loop at higher frequencies

[Jordan Palacios]

Hi.

We've been working with the etherlab master for some time now. On our
current setup we have around 40 slaves and our control loop runs at 1Khz
without any issues. We use the stable version of etherlab with the 20180622
patchset.

Recently we've tried increasing the loop frequency to achieve a better
control. The target frequency is 4Khz. After some troubles we managed a
stable control at 2Khz. Then we went for the 4Khz and this is where we have
hit a roadblock.

A warning like this is generated each second:

[11547.183302] EtherCAT WARNING 0: 4000 datagrams UNMATCHED!
[11547.619399] EtherCAT WARNING: Datagram ffff88040b6bf318 (domain0-0-main)
was SKIPPED 4004 times.

As per what Florian explained here
<http://lists.etherlab.org/pipermail/etherlab-users/2009/000386.html> this
means that the answer to the last datagram sent has not been received yet.
I don't think this is related to the rate at which we execute the control
loop. We have instrumented it and is steady at 250us with a jitter below
10us (see attachment). Furthermore, we have also enabled the ethercat
master debug interface. Here is a sample of the traffic output using tshark:

 7807 0.975750173 MS-NLB-PhysServer-19_95:2e:e1:b0 → Broadcast    ECAT 810
'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 69
 7808 0.975757725 Congatec_2e:e1:b0 → Broadcast    ECAT 810 'LRW': Len:
782, Adp 0x0, Ado 0x0, Wc 0
 7809 0.976000284 MS-NLB-PhysServer-19_95:2e:e1:b0 → Broadcast    ECAT 810
'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 69
 7810 0.976007872 Congatec_2e:e1:b0 → Broadcast    ECAT 810 'LRW': Len:
782, Adp 0x0, Ado 0x0, Wc 0
 7811 0.976252515 MS-NLB-PhysServer-19_95:2e:e1:b0 → Broadcast    ECAT 810
'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 69
 7812 0.976260050 Congatec_2e:e1:b0 → Broadcast    ECAT 810 'LRW': Len:
782, Adp 0x0, Ado 0x0, Wc 0

The master received is done right at the beginning of the loop, and is
followed by the domain process, domain queue and master send. The
calculations are done after. We instrumented the received/send calls and we
know together take around 20us.

We originally made our calculations before the domain queue and master
send. After reading what Graeme explains here
<http://lists.etherlab.org/pipermail/etherlab-users/2018/003351.html> we
changed the order to allow for more time for the data in the wire. This
change helped achieving the 2Khz.

All of this means that the time at which the frames are received is another
good indicator of the correct control loop periodicity. Note also how the
next frame is sent right after.

Thing is, even though the WC is the expected (69 in this case), we think
that these frames are always late. The datagram received is the one of the
previous cycle. Hence the constant warning of 4000 datagrams skipped. The
statistics reported by the "ethercat master" command show 4000 frames
transmitted/received per second and no frames are lost.

Checking the "DC system time transmission delay" of the last slave we get a
value of 38995 ns, which would translate to something like 80us for the
frame to be in the wire. Keeping in mind we are allowing for 230us for the
datagram to return (250us cycle minus 20us for receive/send) it should be
more than enough. Yet something doesn't add up.

The only thing that seems to help is reducing the amount of slaves. If we
switch to something like a 25 slave configuration then we manage to control
at 4KHz without issues. We don't physically disconnect them though so
the DC system time transmission delay remains the same. Obviously frame
size is smaller.

We are not performing any calls related to distributed clocks in the
master. If I understand correctly DC are used for synchronizing the data
processing of the slaves. It should not affect the frame delay, right?

Any ideas are appreciated.

Kind regards.

Jordán.

-- 

Jordán Palacios

Software Engineer

C/ Pujades 77-79, 4-4

08005 Barcelona, Spain

Skype jordanpalacios.pal-robotics

Tel +34 93 414 53 47

*jordan.palacios at pal-robotics.com <jordan.palacios at pal-robotics.com>*

www.pal-robotics.com
Facebook <https://www.facebook.com/palrobotics/> | Twitter
<https://twitter.com/PALRobotics> | YouTube
<http://www.youtube.com/user/PALRobotics> | Blog
<http://blog.pal-robotics.com/>

AVISO DE CONFIDENCIALIDAD: Este mensaje y sus documentos adjuntos, pueden
contener información privilegiada y/o confidencial que está dirigida
exclusivamente a su destinatario.  Si usted recibe este mensaje y no es el
destinatario indicado, o el empleado encargado de su entrega a dicha
persona, por favor, notifíquelo inmediatamente y remita el mensaje original
a la dirección de correo electrónico indicada. Cualquier copia, uso o
distribución no autorizados de esta comunicación queda estrictamente
prohibida.

CONFIDENTIALITY NOTICE: This e-mail and the accompanying document(s) may
contain confidential information which is privileged and intended only for
the individual or entity to whom they are addressed.  If you are not the
intended recipient, you are hereby notified that any disclosure, copying,
distribution or use of this e-mail and/or accompanying document(s) is
strictly prohibited.  If you have received this e-mail in error, please
immediately notify the sender at the above e-mail address.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191030/61d662ae/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jitter.png
Type: image/png
Size: 120942 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191030/61d662ae/attachment-0003.png>