<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-NZ" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">It is sounding like the data time on the wire is taking too long (> 250us).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Besides doubling the "DC system time transmission delay" of the last slave, you can also check the "Diff [ns]" value of your first slave. This may give you a more accurate idea if you have a star
topology, as the return trip of the frame from the last slave bypasses the fingers of the stars. e.g.:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">ethercat slaves -v -p0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">=== Master 0, Slave 0 ===<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">Alias: 10001<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">Device: Main<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">State: OP<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">Flag: +<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">Identity:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> Vendor Id: 0x00000002<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> Product code: 0x04562c52<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> Revision number: 0x00110000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> Serial number: 0x00000000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">DL information:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> FMMU bit operation: no<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> Distributed clocks: yes, 64 bit<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> DC system time transmission delay: 0 ns<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">Port Type Link Loop Signal NextSlave RxTime [ns] Diff [ns] NextDc [ns]<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> 0* EBUS up open yes - 3638440690 0 0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> 1 MII up open yes 1 3638442410
<b><span style="background:yellow;mso-highlight:yellow">1720</span></b> 560<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> 2 N/A down closed no - - - -<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> 3 N/C down closed no - - - -<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">You will also need to add on 2 * "master to first slave transmission delay" which you will probably need to guess. If you use a Beckhoff CX2020 (or similar) where the CX2100 is directly connected
to the EBus this will likely be around 150us. If your master has an ethernet port and the first slave is an amp or a coupler this may be around 550us or more (depending on cable length). You could also calc the frame transmission time. Your frame length
is </span><span style="font-family:"Courier New"">782</span><span style="mso-fareast-language:EN-US"> bytes. So that should take ~ 6 - 7us. Plus whatever other overheads the ethernet card / driver has.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Reducing the number of configured slaves (even with them still physically plugged in) will reduce the ec_master_domain_queue() and ec_master_send() overhead time, getting the frame to the wire quicker.
It will also reduce the frame length and the related frame transmission time (especially if you remove slaves with bigger datagram overhead). It looks like this is enough to result in the total frame roundtrip time being reduce enough for it to return and
be ready by the time that ec_master_receive() is called.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Just FYI, I have a machine downstairs at the moment with 42 slaves (14 of which are amps), linear topology. The last slaves transmission delay is 19560ns. The first slaves Diff is 39390ns. The
EtherCAT frame size is 886 bytes. The ec_master_receive() to ec_master_send() time is approx 23us. So your values sound in line with what I have, except that the wireshark output is showing a large cycle time.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">In general it's looking like the time on the wire is taking longer than it should. How are you capturing the EtherCAT frame data? e.g.: a switch between your master and first slave? If so, try
moving it further down the chain, doing an "ethercat rescan" and check the transmission delays. See if the switch is causing a large delay between those slaves.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">What is your network card and driver used by the master? What is your realtime system? If you are using tshark on the master PC there might be delays being introduced in the network card driver.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Graeme.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> etherlab-users <etherlab-users-bounces@etherlab.org>
<b>On Behalf Of </b>Jordan Palacios<br>
<b>Sent:</b> Thursday, 31 October 2019 4:36 AM<br>
<b>To:</b> etherlab-users@etherlab.org<br>
<b>Subject:</b> [etherlab-users] Control loop at higher frequencies<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We've been working with the etherlab master for some time now. On our current setup we have around 40 slaves and our control loop runs at 1Khz without any issues. We use the stable version of etherlab with the 20180622 patchset.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Recently we've tried increasing the loop frequency to achieve a better control. The target frequency is 4Khz. After some troubles we managed a stable control at 2Khz. Then we went for the 4Khz and this is where we have hit a roadblock.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">A warning like this is generated each second:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">[11547.183302] EtherCAT WARNING 0: 4000 datagrams UNMATCHED!<br>
[11547.619399] EtherCAT WARNING: Datagram ffff88040b6bf318 (domain0-0-main) was SKIPPED 4004 times.</span><o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">As per what Florian explained <a href="http://lists.etherlab.org/pipermail/etherlab-users/2009/000386.html">here</a> this means that the answer to the last datagram sent has not been received yet. I don't think this is related to the rate
at which we execute the control loop. We have instrumented it and is steady at 250us with a jitter below 10us (see attachment). Furthermore, we have also enabled the ethercat master debug interface. Here is a sample of the traffic output using tshark:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> 7807 0.975750173 MS-NLB-PhysServer-19_95:2e:e1:b0 → Broadcast ECAT 810 'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 69
<br>
7808 0.975757725 Congatec_2e:e1:b0 → Broadcast ECAT 810 'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 0
<br>
7809 0.976000284 MS-NLB-PhysServer-19_95:2e:e1:b0 → Broadcast ECAT 810 'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 69
<br>
7810 0.976007872 Congatec_2e:e1:b0 → Broadcast ECAT 810 'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 0
<br>
7811 0.976252515 MS-NLB-PhysServer-19_95:2e:e1:b0 → Broadcast ECAT 810 'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 69
<br>
7812 0.976260050 Congatec_2e:e1:b0 → Broadcast ECAT 810 'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 0 </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The master received is done right at the beginning of the loop, and is followed by the domain process, domain queue and master send. The calculations are done after. We instrumented the received/send calls and we know together take around
20us.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We originally made our calculations before the domain queue and master send. After reading what Graeme explains <a href="http://lists.etherlab.org/pipermail/etherlab-users/2018/003351.html">here</a> we changed the order to allow for more
time for the data in the wire. This change helped achieving the 2Khz.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">All of this means that the time at which the frames are received is another good indicator of the correct control loop periodicity. Note also how the next frame is sent right after.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thing is, even though the WC is the expected (69 in this case), we think that these frames are always late. The datagram received is the one of the previous cycle. Hence the constant warning of 4000 datagrams skipped. The statistics reported
by the "ethercat master" command show 4000 frames transmitted/received per second and no frames are lost.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Checking the "DC system time transmission delay" of the last slave we get a value of 38995 ns, which would translate to something like 80us for the frame to be in the wire. Keeping in mind we are allowing for 230us for the datagram to return
(250us cycle minus 20us for receive/send) it should be more than enough. Yet something doesn't add up.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The only thing that seems to help is reducing the amount of slaves. If we switch to something like a 25 slave configuration then we manage to control at 4KHz without issues. We don't physically disconnect them though so the DC system time
transmission delay remains the same. Obviously frame size is smaller.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We are not performing any calls related to distributed clocks in the master. If I understand correctly DC are used for synchronizing the data processing of the slaves. It should not affect the frame delay, right?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Any ideas are appreciated.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Kind regards.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Jordán.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
</div>
<div>
<div>
<div>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><b><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">Jordán Palacios</span></b><span style="font-size:9.5pt;color:#500050"><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">Software Engineer</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:black"><img border="0" width="128" height="48" style="width:1.3333in;height:.5in" id="_x0000_i1025" src="https://lh6.googleusercontent.com/mgsVJlQ9p5jN1eI-8SP0YOxUhv0BEXrjZFnqqzYgaopfqhinUDbC7abJqhSV9RmgauHBXoff-GMTrJpOL2B9iYbSUgQN3cY8gIFI1XzcMdqZ6nG_Qera_i9qg2VQk35dG0uR01Ut"></span><span style="font-size:9.5pt;color:#500050"><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:black">C/ Pujades 77-79, 4-4</span><span style="font-size:9.5pt;color:#500050"><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:black">08005 Barcelona, Spain</span><span style="font-size:9.5pt;color:#500050"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><b><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:#FF9900">Skype
</span></b><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:black">jordanpalacios.pal-robotics
</span><o:p></o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><b><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:#FF9900">Tel
</span></b><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:black"><a href="tel:%2B34%2093%20414%2053%2047" target="_blank"><span style="color:#1155CC">+34 93 414 53 47</span></a></span><span style="font-size:9.5pt;color:#500050"><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><u><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:#1155CC"><a href="mailto:jordan.palacios@pal-robotics.com" target="_blank">jordan.palacios@pal-robotics.com</a></span></u><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:9.5pt;color:#500050"><a href="http://www.pal-robotics.com/" target="_blank"><span style="font-family:"Tahoma",sans-serif;color:#1155CC">www.pal-robotics.com</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:#500050"><a href="https://www.facebook.com/palrobotics/" target="_blank"><span style="font-size:10.0pt;color:#1155CC">Facebook</span></a></span><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">
| </span><u><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1155CC"><a href="https://twitter.com/PALRobotics" target="_blank"><span style="color:#1155CC">Twitter</span></a></span></u><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">
| </span><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:#500050"><a href="http://www.youtube.com/user/PALRobotics" target="_blank"><span style="font-size:10.0pt;color:#1155CC">YouTube</span></a></span><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">
| </span><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:#500050"><a href="http://blog.pal-robotics.com/" target="_blank"><span style="font-size:10.0pt;color:#1155CC">Blog</span></a></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;color:#500050"><o:p> </o:p></span></p>
</div>
<div>
<p style="display:inline!important"><span style="font-size:7.0pt;color:#1F497D">AVISO DE CONFIDENCIALIDAD: Este mensaje y sus documentos adjuntos, pueden contener información privilegiada y/o confidencial que está dirigida exclusivamente a su destinatario.
Si usted recibe este mensaje y no es el destinatario indicado, o el empleado encargado de su entrega a dicha persona, por favor, notifíquelo inmediatamente y remita el mensaje original a la dirección de correo electrónico indicada. Cualquier copia, uso o distribución
no autorizados de esta comunicación queda estrictamente prohibida.</span><span style="font-size:9.5pt;color:#500050"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.5pt;color:#500050"><o:p> </o:p></span></p>
</div>
<div>
<p><span lang="EN-US" style="font-size:7.0pt;color:#1F497D">CONFIDENTIALITY NOTICE: This e-mail and the accompanying document(s) may contain confidential information which is privileged and intended only for the individual or entity to whom they are addressed.
If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of this e-mail and/or accompanying document(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the
sender at the above e-mail address.</span><span style="font-size:9.5pt;color:#500050"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>