<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-NZ" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi Jordan,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">The GX-JC06 is a 6 port junction. The timings with and without the GX-JC06 look about correct. Just to be sure you know how they work, the EtherCAT frames are passed to each port of the junction
sequentially, not in parallel. It will also pass each port of the junction regardless of whether it is open or not, with a time overhead per port. I don't have the documentation for the GX-JC06, but here is a diagram from the EK1100:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><img width="445" height="403" style="width:4.6354in;height:4.1979in" id="Picture_x0020_8" src="cid:image008.png@01D590A1.4B955570"><span style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">So with your example where one branch is connect direct to the master vs via the GX-JC06 I would expect the time to increase by approx. 620ns per MII port and 135ns per EBUS port (from your info
below). It looks like the junction is made up of two 4 port chips joined by the EBUS ports, something like this:
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="mso-fareast-language:EN-US"><img width="561" height="306" style="width:5.8437in;height:3.1875in" id="Picture_x0020_11" src="cid:image009.png@01D590A3.62449B30"></span><span style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">So you need to add the overhead of:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">- branch slave return to CX-JC06 (1 x 620ns)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">- EBUS to EBUS link (2 x 135)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">- MII ports (8 x 620)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">That adds up to approx. 4610ns extra, totalling 8070ns (within ballpark of your 8220ns value, also based on a few assumptions).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Adding each new branch will add the overhead of the entire branch (the time taken for the frame to get to the last slave of the branch, and return).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">I would expect the a Beckhoff CU1128 to behave very similarly, except that it sounds like the per port overhead is slightly slower (1000ns vs 620ns).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">I'm still picking it's something at the master / device driver end.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Maybe you could check the ethercat version of the e1000e driver is in use correctly. I'm still running kernel 2.6.32 and connect via a CX2100 so not sure if there's any differences to your kernel
and using the e1000e, but from my dmesg I can see:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">EtherCAT: 1 master waiting for devices.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">ec_cx2100: EtherCAT-capable Beckhoff CX2100 EtherCAT Network Driver (Revision 2)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">ec_cx2100: Found EtherCAT Master (0x0014), Revision: 0x0000, txDMA: 0x01, rxDMA: 0x00<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">ec_cx2100: EtherCAT Master, MAC address: 00:01:05:13:7e:d0, Link: Connected<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">EtherCAT: Accepting 00:01:05:13:7E:D0 as main device for master 0.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">I do also have the ec_e1000e driver for the other ports...<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">ec_e1000e: Ethercat-capable Intel(R) PRO/1000 Network Driver - 1.0.2-k2(ethercat)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Doing a "</span><span style="font-family:"Courier New";mso-fareast-language:EN-US">lspci -k</span><span style="mso-fareast-language:EN-US">"</span><span style="mso-fareast-language:EN-US"> also shows
that the ethercat versions of the drivers are correctly loaded:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">02:00.0 Class 0200: 8086:10d3 ec_e1000e<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">03:00.0 Class 0200: 8086:10d3 ec_e1000e<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-family:"Courier New";mso-fareast-language:EN-US">04:00.0 Class ff00: 15ec:5000 ec_cx2100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Not sure what else to check for.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Graeme.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Jordan Palacios <jordan.palacios@pal-robotics.com>
<br>
<b>Sent:</b> Friday, 1 November 2019 1:23 AM<br>
<b>To:</b> Gavin Lambert <gavin.lambert@tomra.com><br>
<b>Cc:</b> Graeme Foot <Graeme.Foot@touchcut.com>; etherlab-users@etherlab.org<br>
<b>Subject:</b> Re: [etherlab-users] Control loop at higher frequencies<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Hi,<br>
<br>
Thanks both of you for your answers. I'll try to address all your points.<br>
<br>
We run our master in a PC with Ubuntu 16.04. Kernel 4.9.115 with the RT_PREEMPT patch. The network driver used by the master is the e1000e. Here are the details of the network card:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> description: Ethernet interface<br>
product: 82579LM Gigabit Network Connection<br>
vendor: Intel Corporation<br>
physical id: 19<br>
bus info: </span><a href="mailto:pci@0000:00:19.0"><span style="font-family:"Courier New"">pci@0000:00:19.0</span></a><span style="font-family:"Courier New""><br>
logical name: eth1<br>
version: 04<br>
serial: 00:13:95:2e:e1:b0<br>
capacity: 1Gbit/s<br>
width: 32 bits<br>
clock: 33MHz<br>
capabilities: bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation<br>
configuration: autonegotiation=on broadcast=yes driver=e1000e driverversion=3.2.6-k firmware=0.15-4 latency=0 link=no multicast=yes port=twisted pair<br>
resources: irq:26 memory:f7f00000-f7f1ffff memory:f7f35000-f7f35fff ioport:f080(size=32)</span><br>
<br>
The slaves are connected in a star topology using an <a href="http://www.ia.omron.com/products/family/3079/feature.html">OMROM GX-JC06 EtherCAT Junction</a>. I didn't mention this in my previous mail but we had detected that this junction does indeed add considerable
delay. For example, if we connect one of the branches directly to the control PC we measure a DC system time transmission delay of 3460ns. When the same branch is connected to the junction with no other slaves reports 8220ns. This gets worse when we start
adding more slaves in different ports until we reach the 38995ns I wrote in my first mail.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Here is the info from the two slaves that comprise the junction:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">=== Master 0, Slave 0 ===<br>
Device: Main<br>
State: PREOP<br>
Flag: +<br>
Identity:<br>
Vendor Id: 0x00000083<br>
Product code: 0x00000064<br>
Revision number: 0x00010000<br>
Serial number: 0x00000000<br>
DL information:<br>
FMMU bit operation: no<br>
Distributed clocks: yes, 64 bit<br>
DC system time transmission delay: 0 ns<br>
Port Type Link Loop Signal NextSlave RxTime [ns] Diff [ns] NextDc [ns]<br>
0* MII up open yes - 1316003530 0 0<br>
1 MII up open yes 8 1316019530 16000 620<br>
2 EBUS up open yes 15 1316043660 40130 135<br>
3 MII up open yes 1 1316011570 8040 640<br>
General:<br>
Group: Junction Slave<br>
Image name: <br>
Order number: GX-JC06(IN,X2,X3)<br>
Device name: GX-JC06(IN,X2,X3)</span><span style="font-family:"Tahoma",sans-serif">�</span><span style="font-family:"Courier New"">@6</span><span style="font-family:"Tahoma",sans-serif">�</span><span style="font-family:"Courier New"">|</span><span style="font-family:"Tahoma",sans-serif">�</span><span style="font-family:"Courier New"">[</span><span style="font-family:"Tahoma",sans-serif">�</span><span style="font-family:"Courier New"">g</span><span style="font-family:"Tahoma",sans-serif">�����</span><span style="font-family:"Courier New"">X</span><span style="font-family:"Tahoma",sans-serif">���</span><span style="font-family:"Courier New"">[</span><span style="font-family:"Tahoma",sans-serif">�</span><span style="font-family:"Courier New"">u<br>
Flags:<br>
Enable SafeOp: no<br>
Enable notLRW: no<br>
Current consumption: 0 mA</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">=== Master 0, Slave 15 ===<br>
Device: Main<br>
State: PREOP<br>
Flag: +<br>
Identity:<br>
Vendor Id: 0x00000083<br>
Product code: 0x00000065<br>
Revision number: 0x00010000<br>
Serial number: 0x00000000<br>
DL information:<br>
FMMU bit operation: no<br>
Distributed clocks: yes, 64 bit<br>
DC system time transmission delay: 16135 ns<br>
Port Type Link Loop Signal NextSlave RxTime [ns] Diff [ns] NextDc [ns]<br>
0* EBUS up open yes 0 1320871430 0 135<br>
1 MII up open yes 24 1320892930 21500 620<br>
2 MII up open yes 35 1320895290 23860 620<br>
3 MII up open yes 16 1320880610 9180 630<br>
General:<br>
Group: Junction Slave<br>
Image name: <br>
Order number: GX-JC06(X4,X5,X6)<br>
Device name: GX-JC06(X4,X5,X6)</span><span style="font-family:"Tahoma",sans-serif">�</span><span style="font-family:"Courier New"">@6</span><span style="font-family:"Tahoma",sans-serif">�</span><span style="font-family:"Courier New"">|</span><span style="font-family:"Tahoma",sans-serif">�</span><span style="font-family:"Courier New"">[</span><span style="font-family:"Tahoma",sans-serif">�</span><span style="font-family:"Courier New"">g</span><span style="font-family:"Tahoma",sans-serif">�����</span><span style="font-family:"Courier New"">X</span><span style="font-family:"Tahoma",sans-serif">���</span><span style="font-family:"Courier New"">[</span><span style="font-family:"Tahoma",sans-serif">�</span><span style="font-family:"Courier New"">u<br>
Flags:<br>
Enable SafeOp: no<br>
Enable notLRW: no<br>
Current consumption: 0 mA</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We are now in the process of switching to a <a href="https://www.beckhoff.com/CU1128/">
Beckhoff CU1128</a>. Their support has assured us that the delay introduced should be around 1us per port used and should remain constant independently of the number of slaves and frame size.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We tried a setup with a switch in the middle to capture the network traffic. Unfortunately we couldn't make it work. The master would simply fail to find any slaves in the network the moment the switch was plugged in. Even when there was
no other PC/laptop connected to the switch to sniff the traffic. We'll give it another go.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We also took additional measures using two <a href="https://www.beckhoff.ee/english.asp?ethercat/fb1111.htm">Beckhoff FB1111-0142</a>. We connected one of them before the switch and the other after the last slave. Then took the measure
of the delay between the signal of the SOF pins. The result was a delay of around 39.2us, something really close to the value reported by the DC system time transmission delay. Using this approach we took measures of different sections of the bus. So now we
have an idea of the average delay introduced by each slave. This further confirmed the delay introduced by the OMROM junction.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The delay introduced by the PC network card is something we definitely haven't taken into account and may explain why we don't receive the frames on time. I don't think it as much as 550us though. If it was it wouldn't work at 2Khz neither.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Kind regards.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Jordán.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, 31 Oct 2019 at 00:07, Gavin Lambert <<a href="mailto:gavin.lambert@tomra.com" target="_blank">gavin.lambert@tomra.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">When you use the master debug interface (or the newly added “ethercat pcap” command, although you won’t have that in your version yet), the receive timestamps are the time that
they are received by the master, which is the time that you called ecrt_master_receive.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">You can get a better understanding of the “real” wire transfer time by connecting a separate monitoring device between the master and first slave – a proper network monitor is ideal,
but you can made do with a standard Ethernet hub/switch with three ports connected (master, first slave, monitoring laptop running Wireshark), although for best results (especially if it’s a Windows laptop) you should disable all the network protocols on the
laptop adapter other than the minimum it needs to do the Wireshark monitoring, so that it doesn’t inadvertently transmit packets of its own. (These won’t hurt the network itself, since both the master and slave will drop non-EtherCAT packets, but it may worsen
the latency.) This will still be somewhat inaccurate since it’s still timestamping when the packet was processed rather than when it was actually received in hardware, but it’s likely to be more accurate than doing it on the master since it’s not waiting
for the cycle delay.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">As Graeme said, you can also put the monitoring device elsewhere in the network (all packets will always pass every point in the network – except the end), although then it becomes
harder to meaningfully use the timestamps.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p><strong><span style="font-family:"Calibri",sans-serif;color:#595959">Gavin Lambert</span></strong><b><span style="color:#595959"><br>
</span></b><span style="color:#595959">Senior Software Developer<o:p></o:p></span></p>
<table class="MsoNormalTable" border="0" cellpadding="0">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt"></td>
</tr>
</tbody>
</table>
<p><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><img border="0" width="360" height="102" style="width:3.75in;height:1.0625in" id="Picture_x0020_1" src="cid:image001.png@01D5909F.79B9FED0" alt="cid:image001.png@01D5909F.79B9FED0"></span><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
</span><a href="http://www.compacsort.com/" target="_blank"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;text-decoration:none"><img border="0" width="33" height="37" style="width:.3437in;height:.3854in" id="Picture_x0020_2" src="cid:image002.png@01D5909F.79B9FED0" alt="TOMRA"></span></a><a href="https://www.facebook.com/Compacsort" target="_blank"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;text-decoration:none"><img border="0" width="35" height="37" style="width:.3645in;height:.3854in" id="Picture_x0020_3" src="cid:image003.png@01D5909F.79B9FED0" alt="Facebook"></span></a><a href="https://www.linkedin.com/company/compac-sorting-equipment/" target="_blank"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;text-decoration:none"><img border="0" width="35" height="37" style="width:.3645in;height:.3854in" id="Picture_x0020_4" src="cid:image004.png@01D5909F.79B9FED0" alt="Linkedin"></span></a><a href="https://vimeo.com/compacsort" target="_blank"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;text-decoration:none"><img border="0" width="37" height="37" style="width:.3854in;height:.3854in" id="Picture_x0020_5" src="cid:image005.png@01D5909F.79B9FED0" alt="Youtube"></span></a><a href="https://twitter.com/compacsort" target="_blank"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;text-decoration:none"><img border="0" width="33" height="37" style="width:.3437in;height:.3854in" id="Picture_x0020_6" src="cid:image006.png@01D5909F.79B9FED0" alt="twitter"></span></a><a href="https://www.instagram.com/compacsort/" target="_blank"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;text-decoration:none"><img border="0" width="33" height="37" style="width:.3437in;height:.3854in" id="Picture_x0020_7" src="cid:image007.png@01D5909F.79B9FED0" alt="instagram"></span></a><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
<p><b><span style="font-size:8.5pt;color:#595959">COMPAC SORTING EQUIPMENT LTD</span></b><span style="font-size:8.5pt;color:#595959"> | 4 Henderson Pl | Onehunga | Auckland 1061 | New Zealand<br>
Switchboard: +64 96 34 00 88 | </span><a href="http://www.tomra.com" target="_blank"><span style="font-size:8.5pt">tomra.com</span></a><span style="font-size:8.5pt;color:#595959">
<o:p></o:p></span></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" style="border-collapse:collapse">
<tbody>
<tr>
<td valign="top" style="border-top:solid #595959 1.0pt;border-left:none;border-bottom:solid #595959 1.0pt;border-right:none;padding:0cm 0cm 0cm 0cm">
<p><span style="font-size:8.5pt;color:#595959;mso-fareast-language:EN-US">The information contained in this communication and any attachment is confidential and may be legally privileged. It should only be read by the person(s) to whom it is addressed. If you
have received this communication in error, please notify the sender and delete the communication.
<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Graeme Foot<br>
<b>Sent:</b> Thursday, 31 October 2019 11:49<br>
<b>To:</b> Jordan Palacios <</span><a href="mailto:jordan.palacios@pal-robotics.com" target="_blank"><span lang="EN-US">jordan.palacios@pal-robotics.com</span></a><span lang="EN-US">><br>
<b>Cc:</b> </span><a href="mailto:etherlab-users@etherlab.org" target="_blank"><span lang="EN-US">etherlab-users@etherlab.org</span></a><span lang="EN-US"><br>
<b>Subject:</b> Re: [etherlab-users] Control loop at higher frequencies</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">It is sounding like the data time on the wire is taking too long (> 250us).<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Besides doubling the "DC system time transmission delay" of the last slave, you can also check the "Diff [ns]" value of your first slave. This may give you a more accurate idea
if you have a star topology, as the return trip of the frame from the last slave bypasses the fingers of the stars. e.g.:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New"">ethercat slaves -v -p0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New"">=== Master 0, Slave 0 ===</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New"">Alias: 10001</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New"">Device: Main</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New"">State: OP</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New"">Flag: +</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New"">Identity:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New""> Vendor Id: 0x00000002</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New""> Product code: 0x04562c52</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New""> Revision number: 0x00110000</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New""> Serial number: 0x00000000</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New"">DL information:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New""> FMMU bit operation: no</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New""> Distributed clocks: yes, 64 bit</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New""> DC system time transmission delay: 0 ns</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New"">Port Type Link Loop Signal NextSlave RxTime [ns] Diff [ns] NextDc [ns]</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New""> 0* EBUS up open yes - 3638440690 0 0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New""> 1 MII up open yes 1 3638442410
<b><span style="background:yellow">1720</span></b> 560</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New""> 2 N/A down closed no - - - -</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-family:"Courier New""> 3 N/C down closed no - - - -</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">You will also need to add on 2 * "master to first slave transmission delay" which you will probably need to guess. If you use a Beckhoff CX2020 (or similar) where the CX2100 is
directly connected to the EBus this will likely be around 150us. If your master has an ethernet port and the first slave is an amp or a coupler this may be around 550us or more (depending on cable length). You could also calc the frame transmission time.
Your frame length is <span style="font-family:"Courier New"">782</span> bytes. So that should take ~ 6 - 7us. Plus whatever other overheads the ethernet card / driver has.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Reducing the number of configured slaves (even with them still physically plugged in) will reduce the ec_master_domain_queue() and ec_master_send() overhead time, getting the frame
to the wire quicker. It will also reduce the frame length and the related frame transmission time (especially if you remove slaves with bigger datagram overhead). It looks like this is enough to result in the total frame roundtrip time being reduce enough
for it to return and be ready by the time that ec_master_receive() is called.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Just FYI, I have a machine downstairs at the moment with 42 slaves (14 of which are amps), linear topology. The last slaves transmission delay is 19560ns. The first slaves Diff
is 39390ns. The EtherCAT frame size is 886 bytes. The ec_master_receive() to ec_master_send() time is approx 23us. So your values sound in line with what I have, except that the wireshark output is showing a large cycle time.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">In general it's looking like the time on the wire is taking longer than it should. How are you capturing the EtherCAT frame data? e.g.: a switch between your master and first
slave? If so, try moving it further down the chain, doing an "ethercat rescan" and check the transmission delays. See if the switch is causing a large delay between those slaves.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">What is your network card and driver used by the master? What is your realtime system? If you are using tshark on the master PC there might be delays being introduced in the network
card driver.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Regards,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Graeme.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> etherlab-users <</span><a href="mailto:etherlab-users-bounces@etherlab.org" target="_blank"><span lang="EN-US">etherlab-users-bounces@etherlab.org</span></a><span lang="EN-US">>
<b>On Behalf Of </b>Jordan Palacios<br>
<b>Sent:</b> Thursday, 31 October 2019 4:36 AM<br>
<b>To:</b> </span><a href="mailto:etherlab-users@etherlab.org" target="_blank"><span lang="EN-US">etherlab-users@etherlab.org</span></a><span lang="EN-US"><br>
<b>Subject:</b> [etherlab-users] Control loop at higher frequencies</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi.<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We've been working with the etherlab master for some time now. On our current setup we have around 40 slaves and our control loop runs at 1Khz without any issues. We use the stable
version of etherlab with the 20180622 patchset.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Recently we've tried increasing the loop frequency to achieve a better control. The target frequency is 4Khz. After some troubles we managed a stable control at 2Khz. Then we went
for the 4Khz and this is where we have hit a roadblock.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">A warning like this is generated each second:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New"">[11547.183302] EtherCAT WARNING 0: 4000 datagrams UNMATCHED!<br>
[11547.619399] EtherCAT WARNING: Datagram ffff88040b6bf318 (domain0-0-main) was SKIPPED 4004 times.</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">As per what Florian explained <a href="https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.etherlab.org%2Fpipermail%2Fetherlab-users%2F2009%2F000386.html&data=02%7C01%7Cgavin.lambert%40tomra.com%7C04fb9669a65e4a8628df08d75d8b6027%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637080725509845944&sdata=zK%2BbQlmlBTBLOP65qEs7UIA6f%2BKC9XH%2FjyFpB7P2vwA%3D&reserved=0" target="_blank">here</a> this
means that the answer to the last datagram sent has not been received yet. I don't think this is related to the rate at which we execute the control loop. We have instrumented it and is steady at 250us with a jitter below 10us (see attachment). Furthermore,
we have also enabled the ethercat master debug interface. Here is a sample of the traffic output using tshark:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New""> 7807 0.975750173 MS-NLB-PhysServer-19_95:2e:e1:b0 → Broadcast ECAT 810 'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 69
<br>
7808 0.975757725 Congatec_2e:e1:b0 → Broadcast ECAT 810 'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 0
<br>
7809 0.976000284 MS-NLB-PhysServer-19_95:2e:e1:b0 → Broadcast ECAT 810 'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 69
<br>
7810 0.976007872 Congatec_2e:e1:b0 → Broadcast ECAT 810 'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 0
<br>
7811 0.976252515 MS-NLB-PhysServer-19_95:2e:e1:b0 → Broadcast ECAT 810 'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 69
<br>
7812 0.976260050 Congatec_2e:e1:b0 → Broadcast ECAT 810 'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 0 </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The master received is done right at the beginning of the loop, and is followed by the domain process, domain queue and master send. The calculations are done after. We instrumented
the received/send calls and we know together take around 20us.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We originally made our calculations before the domain queue and master send. After reading what Graeme explains <a href="https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.etherlab.org%2Fpipermail%2Fetherlab-users%2F2018%2F003351.html&data=02%7C01%7Cgavin.lambert%40tomra.com%7C04fb9669a65e4a8628df08d75d8b6027%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637080725509855946&sdata=LeK76aJegbLDAZvk04IaKzVrtfbjqY9%2FZ%2B9uetwrSTE%3D&reserved=0" target="_blank">here</a> we
changed the order to allow for more time for the data in the wire. This change helped achieving the 2Khz.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">All of this means that the time at which the frames are received is another good indicator of the correct control loop periodicity. Note also how the next frame is sent right after.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thing is, even though the WC is the expected (69 in this case), we think that these frames are always late. The datagram received is the one of the previous cycle. Hence the constant
warning of 4000 datagrams skipped. The statistics reported by the "ethercat master" command show 4000 frames transmitted/received per second and no frames are lost.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Checking the "DC system time transmission delay" of the last slave we get a value of 38995 ns, which would translate to something like 80us for the frame to be in the wire. Keeping
in mind we are allowing for 230us for the datagram to return (250us cycle minus 20us for receive/send) it should be more than enough. Yet something doesn't add up.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The only thing that seems to help is reducing the amount of slaves. If we switch to something like a 25 slave configuration then we manage to control at 4KHz without issues. We
don't physically disconnect them though so the DC system time transmission delay remains the same. Obviously frame size is smaller.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We are not performing any calls related to distributed clocks in the master. If I understand correctly DC are used for synchronizing the data processing of the slaves. It should
not affect the frame delay, right?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Any ideas are appreciated.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Kind regards.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Jordán.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">-- <o:p></o:p></p>
</div>
<div>
<div>
<div>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><b><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">Jordán Palacios</span></b><o:p></o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">Software Engineer</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:black"><img border="0" width="128" height="48" style="width:1.3333in;height:.5in" id="_x0000_i1026" src="https://lh6.googleusercontent.com/mgsVJlQ9p5jN1eI-8SP0YOxUhv0BEXrjZFnqqzYgaopfqhinUDbC7abJqhSV9RmgauHBXoff-GMTrJpOL2B9iYbSUgQN3cY8gIFI1XzcMdqZ6nG_Qera_i9qg2VQk35dG0uR01Ut" alt="https://lh6.googleusercontent.com/mgsVJlQ9p5jN1eI-8SP0YOxUhv0BEXrjZFnqqzYgaopfqhinUDbC7abJqhSV9RmgauHBXoff-GMTrJpOL2B9iYbSUgQN3cY8gIFI1XzcMdqZ6nG_Qera_i9qg2VQk35dG0uR01Ut"></span><o:p></o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:black">C/ Pujades 77-79, 4-4</span><o:p></o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:black">08005 Barcelona, Spain</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><b><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:#FF9900">Skype
</span></b><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:black">jordanpalacios.pal-robotics
</span><o:p></o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><b><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:#FF9900">Tel
</span></b><a href="tel:%2B34%2093%20414%2053%2047" target="_blank"><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:#1155CC">+34 93 414 53 47</span></a><o:p></o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><a href="mailto:jordan.palacios@pal-robotics.com" target="_blank"><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif">jordan.palacios@pal-robotics.com</span></a><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><a href="https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pal-robotics.com%2F&data=02%7C01%7Cgavin.lambert%40tomra.com%7C04fb9669a65e4a8628df08d75d8b6027%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637080725509865938&sdata=ud6L6AE%2BxMtBBfXHGyILhz9G0T9QFaMpscTQgSIk4pg%3D&reserved=0" target="_blank"><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:#1155CC">www.pal-robotics.com</span></a><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2Fpalrobotics%2F&data=02%7C01%7Cgavin.lambert%40tomra.com%7C04fb9669a65e4a8628df08d75d8b6027%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637080725509865938&sdata=sPlDvHQ7mGc3ID8ZWzaaCQPKgYbRTuzEo02bBDDOb1A%3D&reserved=0" target="_blank"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1155CC">Facebook</span></a><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">
| </span><a href="https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FPALRobotics&data=02%7C01%7Cgavin.lambert%40tomra.com%7C04fb9669a65e4a8628df08d75d8b6027%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637080725509875939&sdata=7OB76GKDkYcMFepI3cXw%2FLmIEGVlARXPHuzvES2clmQ%3D&reserved=0" target="_blank"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1155CC">Twitter</span></a><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">
| </span><a href="https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.youtube.com%2Fuser%2FPALRobotics&data=02%7C01%7Cgavin.lambert%40tomra.com%7C04fb9669a65e4a8628df08d75d8b6027%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637080725509875939&sdata=TJA6vduSHtLn93DYqKU3EVEnYUxyZit2U5xCHfhPMdo%3D&reserved=0" target="_blank"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1155CC">YouTube</span></a><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">
| </span><a href="https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fblog.pal-robotics.com%2F&data=02%7C01%7Cgavin.lambert%40tomra.com%7C04fb9669a65e4a8628df08d75d8b6027%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637080725509885933&sdata=Oi%2B7eys7YYEEGhMCvGHtjC3BjZgTXjRzc5Ry7DVikeY%3D&reserved=0" target="_blank"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1155CC">Blog</span></a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt;color:#500050"> </span><o:p></o:p></p>
</div>
<div>
<p><span style="font-size:7.0pt;color:#1F497D">AVISO DE CONFIDENCIALIDAD: Este mensaje y sus documentos adjuntos, pueden contener información privilegiada y/o confidencial que está dirigida exclusivamente a su destinatario. Si usted recibe este mensaje y no
es el destinatario indicado, o el empleado encargado de su entrega a dicha persona, por favor, notifíquelo inmediatamente y remita el mensaje original a la dirección de correo electrónico indicada. Cualquier copia, uso o distribución no autorizados de esta
comunicación queda estrictamente prohibida.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt;color:#500050"> </span><o:p></o:p></p>
</div>
<div>
<p><span lang="EN-US" style="font-size:7.0pt;color:#1F497D">CONFIDENTIALITY NOTICE: This e-mail and the accompanying document(s) may contain confidential information which is privileged and intended only for the individual or entity to whom they are addressed.
If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of this e-mail and/or accompanying document(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the
sender at the above e-mail address.</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><b><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">Jordán Palacios</span></b><span style="font-size:9.5pt;color:#500050"><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:black">Software Engineer</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:black"><img border="0" width="128" height="48" style="width:1.3333in;height:.5in" id="_x0000_i1025" src="https://lh6.googleusercontent.com/mgsVJlQ9p5jN1eI-8SP0YOxUhv0BEXrjZFnqqzYgaopfqhinUDbC7abJqhSV9RmgauHBXoff-GMTrJpOL2B9iYbSUgQN3cY8gIFI1XzcMdqZ6nG_Qera_i9qg2VQk35dG0uR01Ut" alt="https://lh6.googleusercontent.com/mgsVJlQ9p5jN1eI-8SP0YOxUhv0BEXrjZFnqqzYgaopfqhinUDbC7abJqhSV9RmgauHBXoff-GMTrJpOL2B9iYbSUgQN3cY8gIFI1XzcMdqZ6nG_Qera_i9qg2VQk35dG0uR01Ut"></span><span style="font-size:9.5pt;color:#500050"><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:black">C/ Pujades 77-79, 4-4</span><span style="font-size:9.5pt;color:#500050"><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:black">08005 Barcelona, Spain</span><span style="font-size:9.5pt;color:#500050"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><b><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:#FF9900">Skype
</span></b><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:black">jordanpalacios.pal-robotics
</span><o:p></o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><b><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:#FF9900">Tel
</span></b><a href="tel:%2B34%2093%20414%2053%2047" target="_blank"><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:#1155CC">+34 93 414 53 47</span></a><span style="font-size:9.5pt;color:#500050"><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><a href="mailto:jordan.palacios@pal-robotics.com" target="_blank"><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif">jordan.palacios@pal-robotics.com</span></a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><a href="http://www.pal-robotics.com/" target="_blank"><span style="font-size:9.5pt;font-family:"Tahoma",sans-serif;color:#1155CC">www.pal-robotics.com</span></a><span style="font-size:9.5pt;color:#500050"><o:p></o:p></span></p>
<p class="MsoNormal"><a href="https://www.facebook.com/palrobotics/" target="_blank"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1155CC">Facebook</span></a><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">
| </span><a href="https://twitter.com/PALRobotics" target="_blank"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1155CC">Twitter</span></a><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"> |
</span><a href="http://www.youtube.com/user/PALRobotics" target="_blank"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1155CC">YouTube</span></a><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"> |
</span><a href="http://blog.pal-robotics.com/" target="_blank"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:#1155CC">Blog</span></a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;color:#500050"><o:p> </o:p></span></p>
</div>
<div>
<p><span style="font-size:7.0pt;color:#1F497D">AVISO DE CONFIDENCIALIDAD: Este mensaje y sus documentos adjuntos, pueden contener información privilegiada y/o confidencial que está dirigida exclusivamente a su destinatario. Si usted recibe este mensaje y no
es el destinatario indicado, o el empleado encargado de su entrega a dicha persona, por favor, notifíquelo inmediatamente y remita el mensaje original a la dirección de correo electrónico indicada. Cualquier copia, uso o distribución no autorizados de esta
comunicación queda estrictamente prohibida.</span><span style="font-size:9.5pt;color:#500050"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.5pt;color:#500050"><o:p> </o:p></span></p>
</div>
<div>
<p><span lang="EN-US" style="font-size:7.0pt;color:#1F497D">CONFIDENTIALITY NOTICE: This e-mail and the accompanying document(s) may contain confidential information which is privileged and intended only for the individual or entity to whom they are addressed.
If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of this e-mail and/or accompanying document(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the
sender at the above e-mail address.</span><span style="font-size:9.5pt;color:#500050"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>