<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle21
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1271166518;
mso-list-template-ids:-1657506078;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-NZ" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi</span> <span style="mso-fareast-language:EN-US">
Jakub ,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Without looking too deeply, Ethernet over EtherCAT (EoE) requires the ecrt_master_callbacks() method to be used to provide synchronization between the master realtime thread and the EoE thread.
(Your example application does not contain this method.)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">ecrt.h says:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">/** Sets the locking callbacks.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">*<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">* For concurrent master access, i. e. if other instances than the application<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">* want to send and receive datagrams on the bus, the application has to<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">* provide a callback mechanism. This method takes two function pointers as<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">* its parameters. Asynchronous master access (like EoE processing) is only<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">* possible if the callbacks have been set.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">*<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">* The task of the send callback (\a send_cb) is to decide, if the bus is<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">* currently accessible and whether or not to call the ecrt_master_send_ext()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">* method.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">*<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">* The task of the receive callback (\a receive_cb) is to decide, if a call to<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">* ecrt_master_receive() is allowed and to execute it respectively.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">*<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">* \attention This method has to be called before ecrt_master_activate().<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">*/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US">void ecrt_master_callbacks(<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> ec_master_t *master, /**< EtherCAT master */<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> void (*send_cb)(void *), /**< Datagram sending callback. */<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> void (*receive_cb)(void *), /**< Receive callback. */<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> void *cb_data /**< Arbitrary pointer passed to the callback functions.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> */<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";mso-fareast-language:EN-US"> );<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Note: This method needs to be called from a kernel space application, with kernel space callback methods. If you have a user space only application things become a lot more complex.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">If you don’t need EoE, then it’s definitely best to disable and ignore it.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Graeme Foot.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Etherlab-users <etherlab-users-bounces@etherlab.org>
<b>On Behalf Of </b>j.sikorski@utwente.nl<br>
<b>Sent:</b> Thursday, 29 July 2021 03:02<br>
<b>To:</b> etherlab-users@etherlab.org<br>
<b>Subject:</b> Re: [Etherlab-users] Fw: Master malfunctions after a few runs of a program<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div id="divtagdefaultwrapper">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:12.0pt;color:black">Hi everyone,<br>
<br>
I have gone silent in that topic for much longer than expected due to other obligations. However, the problem was becoming more and more crucial to solve, hence I finally managed to find some time for it. I dug in it and eventually found a fix.<br>
<br>
<br>
<b>tl;dr version: </b>the EoE thread was responsible for all the mess I experienced. Recompiling the master with --disable-eoe solved the problem on all affected machines. Precise cause of the problem remains unknown for me, but I won't investigate further,
since my time is limited. If anyone from the Etherlab developers wants to investigate this issue, I will be happy to provide any further info you may need or run whichever test code you may want me to run. Just let me know.<br>
<br>
<br>
<b>Full version: </b><o:p></o:p></span></p>
<ol start="1" type="1">
<li class="MsoNormal" style="color:black;mso-margin-top-alt:auto;margin-bottom:12.0pt;mso-list:l0 level1 lfo1">
<span style="font-size:12.0pt">First, I made a hello world version of my code (main.cpp attached here) as suggested. That didn't manage to solve the issue. The problematic behaviour remained unchanged<o:p></o:p></span></li><li class="MsoNormal" style="color:black;mso-margin-top-alt:auto;margin-bottom:12.0pt;mso-list:l0 level1 lfo1">
<span style="font-size:12.0pt">The thing that helped to some extent was switching from Kernel 4.19 to 4.4. The issue started appearing more seldom, although after some time the master would unavoidably malfunction. The downside was that it also started to appear
regardless whether the application was run for the first or consecutive times.<o:p></o:p></span></li></ol>
<div>
<ol start="3" type="1">
<li class="MsoNormal" style="color:black;mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
<span style="font-size:12.0pt">I thought the problem may be due to the fact that I used -lowlatency kernel instead of RTPREEMPT. I patched and installed 4.4.270-rt222 kernel on Ubuntu 18.04 on one of the affected computers. No change.<o:p></o:p></span></li></ol>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<div>
<ol start="4" type="1">
<li class="MsoNormal" style="color:black;mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
<span style="font-size:12.0pt">At this point I was already planning to write the message to this mailing list asking for any further suggestions. I was copying the output from dmesg after a crash. The output looked like this:<o:p></o:p></span></li></ol>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 213.803906] e1000e: enp3s0f0 NIC Link is Down<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 217.855649] e1000e: enp3s0f0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: None<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 217.855770] e1000e 0000:03:00.0 enp3s0f0: 10/100 speed: disabling TSO<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.034625] EtherCAT: Master driver 1.5.2 365eea9297ef<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.035309] EtherCAT: 1 master waiting for devices.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.211992] e1000e: enp3s0f1 NIC Link is Down<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.403064] e1000e: enp3s0f0 NIC Link is Down<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.440810] e1000e 0000:00:19.0 eno1: removed PHC<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.561885] e1000e: eno1 NIC Link is Down<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.605592] ec_e1000e: EtherCAT-capable Intel(R) PRO/1000 Network Driver - 3.2.6-k-EtherCAT<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.605597] ec_e1000e: Copyright(c) 1999 - 2015 Intel Corporation.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.605851] ec_e1000e 0000:00:19.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.808912] ec_e1000e 0000:00:19.0 eth0: registered PHC clock<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.808919] ec_e1000e 0000:00:19.0 eth0: (PCI Express:2.5GT/s:Width x1) b8:ca:3a:7a:a4:4b<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.808923] ec_e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.808989] ec_e1000e 0000:00:19.0 eth0: MAC: 10, PHY: 11, PBA No: 1011FF-0FF<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.809238] ec_e1000e 0000:03:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.811898] ec_e1000e 0000:00:19.0 eno1: renamed from eth0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.828469] IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 223.982809] EtherCAT: Accepting 00:24:81:81:A6:98 as main device for master 0.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 224.063994] IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 224.127583] EtherCAT 0: Starting EtherCAT-IDLE thread.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 224.127784] ec_e1000e 0000:03:00.0 ecm0 (uninitialized): (PCI Express:2.5GT/s:Width x4) 00:24:81:81:a6:98<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 224.127791] ec_e1000e 0000:03:00.0 ecm0 (uninitialized): Intel(R) PRO/1000 Network Connection<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 224.127876] ec_e1000e 0000:03:00.0 ecm0 (uninitialized): MAC: 0, PHY: 4, PBA No: D51930-006<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 224.128134] ec_e1000e 0000:03:00.1: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 224.302086] ec_e1000e 0000:03:00.1 eth0: (PCI Express:2.5GT/s:Width x4) 00:24:81:81:a6:99<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 224.302093] ec_e1000e 0000:03:00.1 eth0: Intel(R) PRO/1000 Network Connection<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 224.302176] ec_e1000e 0000:03:00.1 eth0: MAC: 0, PHY: 4, PBA No: D51930-006<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 224.309653] ec_e1000e 0000:03:00.1 enp3s0f1: renamed from eth0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 224.338049] IPv6: ADDRCONF(NETDEV_UP): enp3s0f1: link is not ready<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 224.582923] IPv6: ADDRCONF(NETDEV_UP): enp3s0f1: link is not ready<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 225.430594] EtherCAT: Requesting master 0...<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 225.430602] EtherCAT: Successfully requested master 0.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 225.430682] EtherCAT 0: Domain0: Logical address 0x00000000, 15 byte, expected working counter 3.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 225.430686] EtherCAT 0: Datagram domain0-0-main: Logical offset 0x00000000, 15 byte, type LRW at ffff8807efdb9d98.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 225.430705] EtherCAT 0: Master thread exited.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 225.430708] EtherCAT 0: Starting EtherCAT-OP thread.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 226.128409] ec_e1000e: ecm0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: None<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 226.128533] ec_e1000e 0000:03:00.0 ecm0 (uninitialized): 10/100 speed: disabling TSO<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 226.128538] EtherCAT 0: Link state of ecm0 changed to UP.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 226.130421] EtherCAT 0: 4 slave(s) responding on main device.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 226.130427] EtherCAT 0: Slave states on main device: INIT.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 226.136394] EtherCAT 0: Scanning bus.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.306357] EtherCAT 0: eoe0s2 MAC address derived from NIC part of ecm0 MAC address<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.306529] EtherCAT 0-main-2: Linked to EoE handler eoe0s2<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.314849] IPv6: ADDRCONF(NETDEV_UP): eoe0s2: link is not ready<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.345422] ec_e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.345462] IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.406341] EtherCAT 0: eoe0s0 MAC address derived from NIC part of ecm0 MAC address<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.406507] EtherCAT 0-main-0: Linked to EoE handler eoe0s0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.406514] EtherCAT 0: eoe0s1 MAC address derived from NIC part of ecm0 MAC address<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.406614] EtherCAT 0-main-1: Linked to EoE handler eoe0s1<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.427331] EtherCAT 0: Bus scanning completed in 1297 ms.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.427335] EtherCAT 0: Using slave main-0 as DC reference clock.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.427338] EtherCAT 0: Starting EoE thread.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.427391] EtherCAT WARNING 0: No app_time received up to now, abort DC time offset calculation.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.428355] EtherCAT 0: Slave states on main device: PREOP.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.428670] IPv6: ADDRCONF(NETDEV_UP): eoe0s0: link is not ready<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.432557] IPv6: ADDRCONF(NETDEV_UP): eoe0s1: link is not ready<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.434341] EtherCAT 0: Slave states on main device: INIT, PREOP.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.449362] EtherCAT 0: Slave states on main device: PREOP.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.553348] EtherCAT 0: Domain 0: Working counter changed to 2/3.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.554351] EtherCAT 0: Slave states on main device: PREOP, SAFEOP.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.562356] EtherCAT 0: Slave states on main device: PREOP, OP.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.662329] EtherCAT WARNING 0-main-2: Failed to receive mbox check datagram for eoe0s2.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.662334] EtherCAT WARNING 0-main-0: Failed to receive mbox check datagram for eoe0s0.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 227.662336] EtherCAT WARNING 0-main-1: Failed to receive mbox check datagram for eoe0s1.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">(here comes more of similar messages every few seconds, until the moment of crash:)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 471.071642] EtherCAT WARNING 0-main-2: No sending response for eoe0s2 after 100 tries.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 475.148519] EtherCAT 0: Domain 0: Working counter changed to 0/12.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 475.463475] EtherCAT WARNING 0: 1 datagram UNMATCHED!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 475.491858] EtherCAT WARNING: Datagram ffff8807f58aab58 (domain0-0-main) was SKIPPED 1 time.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 475.540531] EtherCAT WARNING 0-main-1: Failed to receive mbox check datagram for eoe0s1.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 476.149554] EtherCAT 0: Domain 0: Working counter changed to 12/12.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">(at this point the master freezes; the slaves go into "EtherCAT communication error" mode and start blinking with red lights like a christmas tree. The master is locked and I need to reset the
computer and slaves to bring the situation back to normal)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Fortunately, I got sidetracked while copying this output. I closed the terminal and did something else for like 3 minutes. When I came back to the issue and ran dmesg again, I was shocked to see
some extra info:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784147] INFO: task EtherCAT-EoE:4601 blocked for more than 120 seconds.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784153] Tainted: G OE 4.4.0-270-rt222 #2<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784154] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784156] EtherCAT-EoE D ffff8807f21ffd40 0 4601 2 0x00000000<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784160] ffff8807f21ffd40 ffff8807f7a28f80 ffff8807f4a45d00 ffff8807f2200000<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784163] ffff8807f4a45d00 ffff8807f4a45d00 7fffffffffffffff ffff8807f679db48<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784166] ffff8807f21ffd60 ffffffff8185551b 7fffffffffffffff ffff8807f679c080<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784169] Call Trace:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784176] [<ffffffff8185551b>] schedule+0x4b/0xe0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784178] [<ffffffff81857f90>] schedule_timeout+0x1c0/0x2f0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784188] [<ffffffffc06cc6db>] ? e1000_intr_msi+0x6b/0x210 [ec_e1000e]<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784191] [<ffffffff81856af6>] __down+0x76/0xc0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784194] [<ffffffff810b0700>] ? __sched_fork+0x1f0/0x230<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784197] [<ffffffff810d8541>] down+0x41/0x50<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784208] [<ffffffffc063b886>] ec_master_eoe_thread+0xb6/0x210 [ec_master]<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784216] [<ffffffffc063b7d0>] ? ecrt_master_reset+0x40/0x40 [ec_master]<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784218] [<ffffffff810aac87>] kthread+0xe7/0x100<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784221] [<ffffffff810aaba0>] ? kthread_worker_fn+0x170/0x170<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784224] [<ffffffff81859ac5>] ret_from_fork+0x55/0x80<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">[ 600.784226] [<ffffffff810aaba0>] ? kthread_worker_fn+0x170/0x170<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">Bingo!!!</span></b><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">5. I reset the computer. Purged the master and reinstalled it ffrom source with the "--disable-eoe" command during configuration. The problem has vanished entirely. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">6. I tested the fix on all affected machines (3 in total) on kernels 4.4-rt, 4.4-lowlatency, and 4.19-lowlatency. It was definitely EoE. Since I don't really need this protocol and don't know much
about it, I didn't investigate any further.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Personally, at this point I am happy to have a functioning EtherCAT stack and I don't need to know, why I experienced this issue. Perhaps it is known to the dev team. However, if that's not the
case and any Etherlab developer wanted to track it further, I can provide all the assistance required. If you need me to collect any info for you or run any test application, just reach out and I will be happy to help. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Once again thank you very much for this amazing piece of software and your assistance with my problem. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Best regards,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Dr. Jakub Sikorski<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Surgical Robotics Laboratory<br>
Horstring W-130<br>
University of Twente<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Drienerlolaan 5<br>
7500AE Enschede<br>
The Netherlands<br>
<a href="mailto:j.sikorski@utwente.nl">j.sikorski@utwente.nl</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><br>
________________________________<br>
From: Sikorski, J. (ET)<br>
Sent: 03 March 2021 00:30:55<br>
To: Richard Hacker; <a href="mailto:etherlab-users@etherlab.org">etherlab-users@etherlab.org</a><br>
Subject: Re: [Etherlab-users] Fw: Master malfunctions after a few runs of a program<br>
<br>
Hi Richard,<br>
<br>
This is actually a stunningly obvious and powerful suggestion that I completely overlooked. Thank you a lot.<br>
I will do exactly that. Should post again within this thread once I get down to it. May take a week or two, as this is not the only project I'm handling atm.<br>
<br>
And honestly, after dealing with the master for years now, I rather expect my code or the hardware (slaves or the network card) to be the source of the problem.<br>
But I will see after I get the "hello world" running.<br>
<br>
Best,<br>
Jakub<br>
________________________________<br>
From: Richard Hacker <<a href="mailto:ha@igh.de">ha@igh.de</a>><br>
Sent: 03 March 2021 00:15:55<br>
To: <a href="mailto:etherlab-users@etherlab.org">etherlab-users@etherlab.org</a><br>
Cc: Sikorski, J. (ET)<br>
Subject: Re: [Etherlab-users] Fw: Master malfunctions after a few runs of a program<br>
<br>
Hi Jakub,<br>
<br>
You seem to be using an application with quite a distributed and<br>
sophisticated functionality -- sorry if I'm treading on your feet there!<br>
<br>
This is certainly necessary when you want to reuse code the way you are<br>
doing, but to debug it when things don't work out becomes increasingly<br>
more difficult. On an philosophical note: nothing is for free ;)<br>
<br>
You say it had been working in a number of previous applications.<br>
Unfortunately even that may go wrong. As Einstein once said:<br>
No number of experiments can prove my theory right, but it takes only<br>
one to prove it wrong!<br>
<br>
What it boils down to is: you will have to reduce your code to a<br>
straight forward, no frills and dingbats, three liner straight forward<br>
C-type hello world:<br>
1) get resources<br>
2) configure<br>
3) activate<br>
4) run in cyclic mode<br>
in the sense of the provided examples (ok that was four lines ;) )<br>
<br>
I am not postulating that the master is exempt from bugs even though it<br>
is running on thousands of applications (see above), but when you're<br>
juggling with C-type pointers you can get silent corruption in places<br>
that you never expected. The fact that the master hangs is certainly not<br>
right, but you'll only find it when one side (your application) is<br>
completely predictable.<br>
<br>
On a side note: recently I was involved in slave development. My master<br>
was also coughing and I thought it was due to my code on the slave. It<br>
boiled down to my network adapter (Intel I219-V) on the master trying to<br>
be clever by bundling frames together before passing them to the kernel<br>
(aka master). A disaster for real time applications! But my three-liner<br>
helped reducing the possible failure space.<br>
<br>
Happy coding!<br>
Richard<br>
<br>
On 2021-03-01 21:38, <a href="mailto:j.sikorski@utwente.nl">j.sikorski@utwente.nl</a> wrote:<br>
> (apologies if it happens to get posted it multiple times, an earlier<br>
> version of this message was seemingly cancelled by me while waiting for<br>
> moderation)<br>
><br>
> Hi everyone,<br>
><br>
><br>
> For a few years now I have been using the Etherlab master stack with<br>
> Gavin Lambert's patchset to drive embedded systems for my research. It's<br>
> an amazing piece of software and I am truly impressed by this work. For<br>
> my applications I have a thin C++ wrapper that allows me to manage the<br>
> ECRT code through classes and reuse it depending on the project. I coded<br>
> the wrapper myself. I'm not fully sure if it is correct, but it has been<br>
> working fine so far...<br>
><br>
><br>
> However, recently, I have installed the stack on a new computer with<br>
> Ubuntu 18.04, Kernel 4.19.177-0419177-lowlatency. I applied all the<br>
> patches from GL and use ec_e1000e driver with a supposedly compatible<br>
> Intel Card (Model EXPI9402PT). I use the stack to drive Technosoft<br>
> Motion iPos4808 BX-CAT drives.<br>
><br>
><br>
> Once I run my code *for the first time*, everything seems to work fine<br>
> until the application exits. Then one of three things can happen:<br>
><br>
><br>
> Either:<br>
><br>
> 1. I can run the application again from start to the end, no issue.<br>
> 2. The application stops working during either<br>
> ecrt_master_activate(master) or ecrt_request_master(0). Neither of<br>
> these functions returns and the code just hangs there.<br>
> 3. The communication works fine for some time, then starts acting up.<br>
> It stops working entirely during the operation (suddenly the data<br>
> I'm receiving ends up corrupted) or while exiting (the code hangs, I<br>
> think also on ecrt_master_deactivate(master) or ecrt_release_master(0)).<br>
><br>
><br>
> When the application malfunctions, I can only force quit it. If I do, I<br>
> have to reboot the computer, as, whatever ethercat-related I do next, I<br>
> receive the error that the module ec_e1000 is still in use.<br>
><br>
> Dmesg from a normal run:<br>
><br>
> [ 1166.127739] EtherCAT: Requesting master 0...<br>
> [ 1166.127743] EtherCAT: Successfully requested master 0.<br>
> [ 1166.127980] EtherCAT 0: Domain0: Logical address 0x00000000, 21 byte,<br>
> expected working counter 3.<br>
> [ 1166.127982] EtherCAT 0: Datagram domain0-0-main: Logical offset<br>
> 0x00000000, 21 byte, type LRW at 0000000098dbfebf.<br>
> [ 1166.128029] EtherCAT 0: Master thread exited.<br>
> [ 1166.128030] EtherCAT 0: Stopping EoE thread.<br>
> [ 1166.128052] EtherCAT 0: EoE thread exited.<br>
> [ 1166.128053] EtherCAT 0: Starting EoE thread.<br>
> [ 1166.128382] EtherCAT 0: Starting EtherCAT-OP thread.<br>
> [ 1166.131218] EtherCAT WARNING 0: No app_time received up to now, abort<br>
> DC time offset calculation.<br>
> [ 1166.176215] EtherCAT 0: Slave states on main device: INIT.<br>
> [ 1166.200217] EtherCAT 0: Slave states on main device: PREOP.<br>
> [ 1166.443626] EtherCAT 0: Domain 0: Working counter changed to 2/3.<br>
> [ 1166.456205] EtherCAT 0: Slave states on main device: SAFEOP.<br>
> [ 1166.464205] EtherCAT 0: Slave states on main device: OP.<br>
> [ 1167.128196] EtherCAT WARNING 0: 2 datagrams UNMATCHED!<br>
> [ 1167.131459] EtherCAT WARNING: Datagram 0000000098dbfebf<br>
> (domain0-0-main) was SKIPPED 2 times.<br>
> [ 1167.447503] EtherCAT 0: Domain 0: 3 working counter changes - now 3/3.<br>
> [ 1170.264093] EtherCAT 0: Domain 0: Working counter changed to 0/3.<br>
> (This continues until the end)<br>
> [ 1193.785359] EtherCAT 0: Domain 0: Working counter changed to 0/3.<br>
> [ 1193.894348] EtherCAT 0: Slave states on main device: SAFEOP + ERROR.<br>
> [ 1194.793518] EtherCAT 0: Master thread exited.<br>
> [ 1194.793520] EtherCAT 0: Stopping EoE thread.<br>
> [ 1194.793538] EtherCAT 0: EoE thread exited.<br>
> [ 1194.793547] EtherCAT 0: Starting EoE thread.<br>
> [ 1194.793674] EtherCAT 0: Starting EtherCAT-IDLE thread.<br>
> [ 1194.793856] EtherCAT 0: 0 slave(s) responding on main device.<br>
> [ 1194.795324] EtherCAT ERROR 0-main-0: AL status message 0x001B: "Sync<br>
> manager watchdog".<br>
> [ 1194.797319] EtherCAT 0: 1 slave(s) responding on main device.<br>
> [ 1194.797321] EtherCAT 0: Slave states on main device: SAFEOP + ERROR.<br>
> [ 1194.799335] EtherCAT 0-main-0: Acknowledged state SAFEOP.<br>
> [ 1194.801321] EtherCAT 0: Slave states on main device: SAFEOP.<br>
> [ 1194.805320] EtherCAT 0: Slave states on main device: INIT.<br>
> [ 1194.817333] EtherCAT 0: Slave states on main device: PREOP.<br>
> [ 1195.281308] EtherCAT WARNING: Datagram 00000000f3c9ebae (master-fsm)<br>
> was SKIPPED 1 time.<br>
> [ 1195.793926] EtherCAT 0: Releasing master...<br>
> [ 1195.793929] EtherCAT 0: Released.<br>
> [ 1195.797305] EtherCAT 0: Stopping EoE thread.<br>
> [ 1195.797324] EtherCAT 0: EoE thread exited.<br>
> [ 1195.810428] EtherCAT 0: Scanning bus.<br>
> [ 1196.038438] EtherCAT 0: eoe0s0 MAC address derived from NIC part of<br>
> ecm0 MAC address<br>
> [ 1196.038615] EtherCAT 0-main-0: Linked to EoE handler eoe0s0<br>
> [ 1196.038620] EtherCAT 0: Bus scanning completed in 241 ms.<br>
> [ 1196.038622] EtherCAT 0: Using slave main-0 as DC reference clock.<br>
> [ 1196.038623] EtherCAT 0: Starting EoE thread.<br>
> [ 1196.049270] EtherCAT 0: Slave states on main device: INIT.<br>
> [ 1196.050455] IPv6: ADDRCONF(NETDEV_UP): eoe0s0: link is not ready<br>
> [ 1196.061266] EtherCAT 0: Slave states on main device: PREOP.<br>
><br>
> Dmesg from a "normal" run that precedes to crash:<br>
><br>
> [ 3289.782513] EtherCAT: Requesting master 0...<br>
> [ 3289.969220] EtherCAT 0: eoe0s0 MAC address derived from NIC part of<br>
> ecm0 MAC address<br>
> [ 3289.970078] EtherCAT 0-main-0: Linked to EoE handler eoe0s0<br>
> [ 3289.970087] EtherCAT 0: Bus scanning completed in 229 ms.<br>
> [ 3289.970089] EtherCAT 0: Using slave main-0 as DC reference clock.<br>
> [ 3289.970091] EtherCAT 0: Starting EoE thread.<br>
> [ 3289.970100] EtherCAT: Successfully requested master 0.<br>
> [ 3289.970321] EtherCAT 0: Domain0: Logical address 0x00000000, 21 byte,<br>
> expected working counter 3.<br>
> [ 3289.970323] EtherCAT 0: Datagram domain0-0-main: Logical offset<br>
> 0x00000000, 21 byte, type LRW at 00000000e95eb248.<br>
> [ 3289.970341] EtherCAT 0: Master thread exited.<br>
> [ 3289.970342] EtherCAT 0: Stopping EoE thread.<br>
> [ 3289.970349] EtherCAT 0: EoE thread exited.<br>
> [ 3289.970350] EtherCAT 0: Starting EoE thread.<br>
> [ 3289.970386] EtherCAT 0: Starting EtherCAT-OP thread.<br>
> [ 3289.977769] EtherCAT WARNING 0: No app_time received up to now, abort<br>
> DC time offset calculation.<br>
> [ 3289.985470] IPv6: ADDRCONF(NETDEV_UP): eoe0s0: link is not ready<br>
> [ 3289.985571] IPv6: ADDRCONF(NETDEV_UP): eoe0s0: link is not ready<br>
> [ 3289.991646] IPv6: ADDRCONF(NETDEV_UP): eoe0s0: link is not ready<br>
> [ 3289.995472] IPv6: ADDRCONF(NETDEV_UP): eoe0s0: link is not ready<br>
> [ 3290.019763] EtherCAT 0: Slave states on main device: INIT.<br>
> [ 3290.043756] EtherCAT 0: Slave states on main device: PREOP.<br>
> [ 3290.270841] EtherCAT 0: Domain 0: Working counter changed to 3/3.<br>
> [ 3290.279764] EtherCAT 0: Slave states on main device: SAFEOP.<br>
> [ 3290.287760] EtherCAT 0: Slave states on main device: OP.<br>
> [ 3290.651875] ec_e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex,<br>
> Flow Control: None<br>
> [ 3290.651920] IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready<br>
> [ 3290.969766] EtherCAT WARNING 0: 5 datagrams UNMATCHED!<br>
> [ 3290.974674] EtherCAT WARNING: Datagram 00000000e95eb248<br>
> (domain0-0-main) was SKIPPED 5 times.<br>
> [ 3290.976777] IPv6: ADDRCONF(NETDEV_CHANGE): eoe0s0: link becomes ready<br>
> [ 3291.274830] EtherCAT 0: Domain 0: 12 working counter changes - now 3/3.<br>
> [ 3291.972739] EtherCAT WARNING 0: 5 datagrams UNMATCHED!<br>
> [ 3291.974880] EtherCAT WARNING: Datagram 00000000e95eb248<br>
> (domain0-0-main) was SKIPPED 5 times.<br>
> [ 3292.094807] EtherCAT WARNING 0-main-0: No sending response for eoe0s0<br>
> after 100 tries.<br>
> [ 3292.278983] EtherCAT 0: Domain 0: 2 working counter changes - now 3/3.<br>
> [ 3292.990788] EtherCAT WARNING 0-main-0: No sending response for eoe0s0<br>
> after 100 tries.<br>
> [ 3294.103893] EtherCAT WARNING 0-main-0: No sending response for eoe0s0<br>
> after 100 tries.<br>
> [ 3294.887689] EtherCAT 0: Domain 0: Working counter changed to 0/3.<br>
> [ 3294.973683] EtherCAT WARNING 0: 2 datagrams UNMATCHED!<br>
> [ 3294.984490] EtherCAT WARNING: Datagram 00000000e95eb248<br>
> (domain0-0-main) was SKIPPED 2 times.<br>
> [ 3295.891819] EtherCAT 0: Domain 0: 15 working counter changes - now 3/3.<br>
> [ 3295.973667] EtherCAT WARNING 0: 5 datagrams UNMATCHED!<br>
> [ 3295.987844] EtherCAT WARNING: Datagram 00000000e95eb248<br>
> (domain0-0-main) was SKIPPED 5 times.<br>
> [ 3298.943805] EtherCAT 0: Domain 0: Working counter changed to 0/3.<br>
> [ 3299.944793] EtherCAT 0: Domain 0: 13 working counter changes - now 3/3.<br>
> [ 3299.976592] EtherCAT WARNING 0: 6 datagrams UNMATCHED!<br>
> [ 3300.000785] EtherCAT WARNING: Datagram 00000000e95eb248<br>
> (domain0-0-main) was SKIPPED 6 times.<br>
> [ 3300.538790] EtherCAT WARNING 0-main-0: No sending response for eoe0s0<br>
> after 100 tries.<br>
> [ 3303.009524] EtherCAT 0: Domain 0: Working counter changed to 0/3.<br>
> [ 3303.980507] EtherCAT WARNING 0: 9 datagrams UNMATCHED!<br>
> [ 3304.013729] EtherCAT WARNING: Datagram 00000000e95eb248<br>
> (domain0-0-main) was SKIPPED 9 times.<br>
> [ 3304.013732] EtherCAT 0: Domain 0: 17 working counter changes - now 3/3.<br>
> [ 3304.981498] EtherCAT WARNING 0: 1 datagram UNMATCHED!<br>
> [ 3305.017954] EtherCAT WARNING: Datagram 00000000e95eb248<br>
> (domain0-0-main) was SKIPPED 1 time.<br>
> [ 3305.017956] EtherCAT 0: Domain 0: 2 working counter changes - now 3/3.<br>
> [ 3306.726450] EtherCAT 0: Domain 0: Working counter changed to 0/3.<br>
> [ 3306.981453] EtherCAT WARNING 0: 6 datagrams UNMATCHED!<br>
> [ 3307.022444] EtherCAT WARNING: Datagram 00000000e95eb248<br>
> (domain0-0-main) was SKIPPED 7 times.<br>
> [ 3307.730694] EtherCAT 0: Domain 0: 19 working counter changes - now 3/3.<br>
> [ 3307.982442] EtherCAT WARNING 0: 4 datagrams UNMATCHED!<br>
> [ 3308.026763] EtherCAT WARNING: Datagram 00000000e95eb248<br>
> (domain0-0-main) was SKIPPED 3 times.<br>
> [ 3310.479387] EtherCAT 0: Domain 0: Working counter changed to 0/3.<br>
> [ 3310.983378] EtherCAT WARNING 0: 5 datagrams UNMATCHED!<br>
> [ 3311.035400] EtherCAT WARNING: Datagram 00000000e95eb248<br>
> (domain0-0-main) was SKIPPED 5 times.<br>
> [ 3311.141385] EtherCAT WARNING 0-main-0: Failed to receive mbox check<br>
> datagram for eoe0s0.<br>
> [ 3311.274500] EtherCAT WARNING 0-main-0: No sending response for eoe0s0<br>
> after 100 tries.<br>
> [ 3311.483392] EtherCAT 0: Domain 0: 11 working counter changes - now 3/3.<br>
> [ 3314.532303] EtherCAT 0: Domain 0: Working counter changed to 0/3.<br>
> [ 3314.986298] EtherCAT WARNING 0: 3 datagrams UNMATCHED!<br>
> [ 3315.048313] EtherCAT WARNING: Datagram 00000000e95eb248<br>
> (domain0-0-main) was SKIPPED 5 times.<br>
> [ 3315.536416] EtherCAT 0: Domain 0: 13 working counter changes - now 3/3.<br>
> [ 3315.989282] EtherCAT WARNING 0: 2 datagrams UNMATCHED!<br>
> [ 3317.715249] EtherCAT 0: Slave states on main device: SAFEOP + ERROR.<br>
> [ 3318.613273] EtherCAT 0: Master thread exited.<br>
> [ 3318.613275] EtherCAT 0: Stopping EoE thread.<br>
> [ 3318.613343] EtherCAT 0: EoE thread exited.<br>
> [ 3318.613353] EtherCAT 0: Starting EoE thread.<br>
> [ 3318.613500] EtherCAT 0: Starting EtherCAT-IDLE thread.<br>
> [ 3318.613614] EtherCAT 0: 0 slave(s) responding on main device.<br>
> [ 3318.616245] EtherCAT ERROR 0-main-0: AL status message 0x001B: "Sync<br>
> manager watchdog".<br>
> [ 3318.618242] EtherCAT 0: 1 slave(s) responding on main device.<br>
> [ 3318.618245] EtherCAT 0: Slave states on main device: SAFEOP + ERROR.<br>
> [ 3318.620256] EtherCAT 0-main-0: Acknowledged state SAFEOP.<br>
> [ 3318.622244] EtherCAT 0: Slave states on main device: SAFEOP.<br>
> [ 3318.627235] EtherCAT 0: Slave states on main device: INIT.<br>
> [ 3318.639238] EtherCAT 0: Slave states on main device: PREOP.<br>
> [ 3318.783218] EtherCAT WARNING: Datagram 0000000019626fd7 (master-fsm)<br>
> was SKIPPED 1 time.<br>
> [ 3319.613733] EtherCAT 0: Releasing master...<br>
> [ 3319.613737] EtherCAT 0: Released.<br>
> [ 3319.615206] EtherCAT 0: Stopping EoE thread.<br>
> (Here Dmesg stream breaks, so maybe it's something related to the EoE<br>
> thread?)<br>
><br>
><br>
> Dmesg of a run that hangs forever:<br>
> [ 3363.180585] EtherCAT: Requesting master 0...<br>
> [ 3365.930011] EtherCAT 0: Waiting for slave scan interrupted by signal.<br>
> [ 3365.930023] EtherCAT ERROR 0: Failed to enter OPERATION phase!<br>
> [ 3370.924180] ec_e1000e: enp3s0f1 NIC Link is Down<br>
> [ 3370.942351] EtherCAT 0: Stopping EoE thread.<br>
> (Again, the last command is "Stopping EoE thread", but the application<br>
> hangs forever at that point.)<br>
><br>
><br>
> The way I start and end the Ethercat master operation is equivalent to<br>
> what I found in the examples, but I distributed and encapsulated it in a<br>
> few classes for reuse.<br>
> The code I use to initialize the master:<br>
><br>
> int EthercatBus::begin_operation()<br>
> {<br>
> if(!network_operating)<br>
> {<br>
> //ECRT Related<br>
><br>
> if(sc.size())<br>
> {<br>
> mass_configure_DOs(); // This function configures SDOs/PDOs<br>
> }<br>
> ecrt_master_activate(master);<br>
> datagram_begin = ecrt_domain_data(domain1);<br>
><br>
> //Initialize the EtherCAT thread<br>
><br>
><br>
> ethercat_thread=make_shared<boost::thread>(&EthercatBus::function_ethercat_thread,this);<br>
> //Cyclic task starts here<br>
> network_operating = 1;<br>
> cout << "Ethercat Network Operational"<< endl;<br>
><br>
><br>
> if(!slaves_configured && sc.size())<br>
> {<br>
> switch_on_slaves(); //This function pushes each slave through<br>
> slave-specific initialization using PDOs + Controlword<br>
> slaves_configured = 1;<br>
> }<br>
> else if(!sc.size())<br>
> {<br>
> slaves_configured = 1;<br>
> }<br>
> }<br>
><br>
> cout << "Slaves On"<< endl;<br>
> }<br>
><br>
><br>
><br>
> The code which I use to deinitialize and release the master is here:<br>
><br>
> int EthercatBus::end_operation()<br>
> {<br>
> if(network_operating)<br>
> {<br>
><br>
> cout << "Shutting down network" << endl;<br>
><br>
> for (vector<shared_ptr<EthercatSlave>>::iterator<br>
> iter=sc.begin(); iter != sc.end(); iter++)<br>
> {<br>
><br>
> int locA = (*iter)->alias;<br>
> int locPos = (*iter)->position;<br>
> cout << "Shutting down slave : " << locA << "/" << locPos<br>
> << endl;<br>
><br>
> (*iter)->switch_off(); //Slave specific deinitialization<br>
> procedure. Cyclic tasks still run at this point.<br>
><br>
> cout << "Slave : " << locA << "/" << locPos << " Shut Down"<br>
> << endl;<br>
> }<br>
><br>
> usleep (1e6);<br>
><br>
><br>
> ethercat_thread->interrupt(); // Cyclic task stopped<br>
> ethercat_thread->join();<br>
> ethercat_thread.reset();<br>
><br>
> usleep (1e6);<br>
><br>
><br>
> cout << "Thread Returned" << endl;<br>
> ecrt_master_deactivate(master);<br>
> cout << "EthercatBus - Master Deactivated" << endl;<br>
> usleep (1e6);<br>
> ecrt_release_master(master);<br>
><br>
><br>
> cout << "EthercatBus - Master Released" << endl;<br>
><br>
> for (vector<shared_ptr<EthercatSlave>>::iterator<br>
> iter=sc.begin(); iter != sc.end();)<br>
> {<br>
> sc.erase(iter);<br>
> }<br>
><br>
> cout << "Ethercat Network shut down" << endl;<br>
> network_operating=0;<br>
> }<br>
> }<br>
><br>
> The most non-standard thing in my code, which may be the source of an<br>
> error, is definitely the use of a PI controller for timing inside a<br>
> cyclic task:<br>
><br>
> while(!boost::this_thread::interruption_requested()) //Cycle until<br>
> the thread is interrupted.<br>
> {<br>
><br>
> cycle_frames_receive();<br>
><br>
> t2=t1; //Store the beginning time of the previous iteration.<br>
> t1 = high_resolution_clock::now();<br>
> tdiff=duration_cast<microseconds>(t1-t2).count();<br>
><br>
> local_counter++;<br>
><br>
> if(local_counter==frequency) //Once every second<br>
> {<br>
><br>
> local_counter=0;<br>
> }<br>
><br>
><br>
> cycle_frames_send();<br>
><br>
> //PI control over the frame period. Robust up to 10 [kHz]<br>
><br>
> t_error_P=(1e6/frequency - tdiff); // Proportional Error<br>
> t_error_I= t_error_I + t_error_P; //Integral Error<br>
> msec = msec + gain_P*(t_error_P) + gain_I*t_error_I //;<br>
> Closing the control;<br>
><br>
> if(first_iter)<br>
> {<br>
> msec=1e6/frequency;<br>
> first_iter=false;<br>
> }<br>
><br>
> if (msec>0){usleep(msec);} //Saturation, as you cannot wait<br>
> less than 0 microseconds.<br>
><br>
> }<br>
><br>
> instead of a regular timer, but this way has always provided me with a<br>
> more exact and stable performance.<br>
><br>
> *If full source code is required, I can supply it. *<br>
> *<br>
> *<br>
> The entire stack so far worked on 5+ different computers. Similar issues<br>
> _were_ happening in the past sometimes, but like once every 1000 runs<br>
> and always while exiting the application (especially in a forceful<br>
> manner, never during the cyclic communication). On this particular<br>
> machine the master usually gets messed up after 2/3 runs.<br>
><br>
> Besides iPOS drives, I work also with Copley Controls Xenus drives. I<br>
> have never had any issues with those. Applications with them run<br>
> smoothly (so maybe it's the problem with Technosof slaves?).<br>
> *<br>
> *<br>
> *I thought it is a problem of the ec_e1000e driver, but the issue<br>
> persists when I use ec_generic driver. *<br>
><br>
> Has anyone ever encountered a similar problem? I have zero knowledge of<br>
> how the ecrt stack is coded internally. If anyone could tell me what I<br>
> am doing wrong, or at least point me towards any way, in which I could<br>
> establish that, I would be extremely obliged.<br>
><br>
> Yours faithfully,<br>
> Dr. Jakub Sikorski<br>
><br>
> Surgical Robotics Laboratory<br>
> Horstring W-130<br>
> University of Twente<br>
> 7500AE Enschede<br>
> The Netherlands<br>
> <a href="mailto:j.sikorski@utwente.nl">j.sikorski@utwente.nl</a><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
<br>
Mit freundlichem Gruß<br>
<br>
Richard Hacker<br>
<br>
--<br>
------------------------------------------------------------------------<br>
<br>
Richard Hacker M.Sc.<br>
<a href="mailto:richard.hacker@igh.de">richard.hacker@igh.de</a><br>
Tel.: +49 201 / 36014-16<br>
<br>
Ingenieurgemeinschaft IgH<br>
Gesellschaft für Ingenieurleistungen mbH<br>
Nordsternstraße 66<br>
D-45329 Essen<br>
<br>
Amtsgericht Essen HRB 11500<br>
USt-Id.-Nr.: DE 174 626 722<br>
Geschäftsführung:<br>
- Dr.-Ing. Siegfried Rotthäuser<br>
- Dr. Sven Beermann, Prokurist<br>
Tel.: +49 201 / 360-14-0<br>
<a href="http://www.igh.de">http://www.igh.de</a><br>
<br>
------------------------------------------------------------------------<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</body>
</html>