For reference, I was able to get EoE working without hang-up by installing rt preempt and rebuilding the EtherCAT Master. Many thanks to Matthieu and Frank for all you help and tips.<div><br></div><div>Attached is the C code I used, if anyone is interested (or a newcomer reading this in the archives). I'll copy and paste the relevant parts below. Almost all of it is pulled from an example somewhere.</div>
<div><br></div><div><br></div>
<div><div> </div><div>#include <linux/version.h></div><div>#include <linux/module.h></div><div>#include <linux/timer.h></div><div>#include <linux/interrupt.h></div><div>#include <linux/err.h></div>
<div>#include <linux/semaphore.h></div><div>#include <linux/time.h></div><div><br></div><div>#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 27)</div><div>#include <linux/semaphore.h></div><div>#else</div>
<div>#include <asm/semaphore.h></div><div>#endif</div></div><div><div><br></div><div>#define PFX "durability.c: "</div><div><br></div><div>#include "/home/tbj/srcroot/ethercat/include/ecrt.h" // EtherCAT realtime interface</div>
<div> </div><div>/*</div><div> * Import the bus topology generated by EtherLAB</div><div> */</div><div>#include "2369topology.h"</div></div><div><br></div><div><br></div><div><div>/*</div><div> * Application Parameters</div>
<div> */</div><div>#define FREQUENCY 2000</div></div><div><br></div><div><br></div><div>...</div><div>...</div><div><br></div><div><div>void </div><div>cyclic_task(unsigned long data)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>{</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>// receive process data</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>down(&master_sem);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>ecrt_master_receive(master);</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>ecrt_domain_process(domain);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>up(&master_sem);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>ktime_get_ts(&lastCycle_);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>// check process data state (optional)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>check_domain_state();</div><div><br></div><div>
<span class="Apple-tab-span" style="white-space:pre"> </span>if (counter) </div><div><span class="Apple-tab-span" style="white-space:pre"> </span>{</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>counter--;</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>} </div><div><span class="Apple-tab-span" style="white-space:pre"> </span>else </div><div><span class="Apple-tab-span" style="white-space:pre"> </span>{ </div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>counter = FREQUENCY;</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>// check for master state</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>check_master_state();</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>}</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>// send process data</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>down(&master_sem);</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>ecrt_domain_queue(domain);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>ecrt_master_send(master);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>up(&master_sem);</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>// restart timer</div><div> timer.expires += HZ / FREQUENCY;</div><div> add_timer(&timer);<span class="Apple-tab-span" style="white-space:pre"> </span></div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>}</div></div><div><br></div><div><div>struct timespec lastCycle_;</div><div>struct timespec lastSend_;</div><div>struct timespec lastRead_;</div><div>struct timespec interval; </div>
</div><div><br></div><div><div>void </div><div>cyclic_task(unsigned long data)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>{</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>// receive process data</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>down(&master_sem);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>ecrt_master_receive(master);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>ecrt_domain_process(domain);</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>up(&master_sem);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>ktime_get_ts(&lastCycle_);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><br></div><div> ...</div><div> ...</div>
<div><br></div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>// send process data</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>down(&master_sem);</div><div>
<span class="Apple-tab-span" style="white-space:pre"> </span>ecrt_domain_queue(domain);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>ecrt_master_send(master);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>up(&master_sem);</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>// restart timer</div><div> timer.expires += HZ / FREQUENCY;</div><div> add_timer(&timer);<span class="Apple-tab-span" style="white-space:pre"> </span></div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>}</div></div><div><br></div><div><br></div><div><br></div><div><div>void </div><div>send_callback(void *cb_data)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>{</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>struct timespec delta = timespec_sub(lastCycle_, lastSend_);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>ec_master_t *m = NULL;</div>
<div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>if (timespec_compare(&delta, &interval) > 0)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>{</div><div>
<br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>m = (ec_master_t *) cb_data;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>down(&master_sem);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>ecrt_master_send_ext(m);</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>up(&master_sem);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>ktime_get_ts(&lastSend_);</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>}</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>}</div><div><br></div><div>void </div><div>receive_callback(void *cb_data)</div><div>
<span class="Apple-tab-span" style="white-space:pre"> </span>{</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>struct timespec delta = timespec_sub(lastCycle_, lastRead_);</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>ec_master_t *m = NULL;</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>if (timespec_compare(&delta, &interval) > 0)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>{</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>m = (ec_master_t *) cb_data;</div>
<div> </div><div><span class="Apple-tab-span" style="white-space:pre"> </span>down(&master_sem);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>ecrt_master_receive(m);<span class="Apple-tab-span" style="white-space:pre"> </span></div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>up(&master_sem);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>ktime_get_ts(&lastRead_);</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>}</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>}</div></div><div><br></div><div><br></div><div><div>int </div><div>__init init_mini_module(void)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>{</div><div> ec_slave_config_t *sc;</div><div> int ret = -1;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><br></div>
<div> master = ecrt_request_master(0);</div><div> if (!master)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>{</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>printk(KERN_INFO PFX "Could not connect to the master service. Exiting.\n\n");</div>
<div> ret = -EBUSY; </div><div><span class="Apple-tab-span" style="white-space:pre"> </span>goto out_return;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>}</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>ktime_get_ts(&lastCycle_);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>ktime_get_ts(&lastSend_);</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>ktime_get_ts(&lastRead_);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>// setup a comparison</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>interval.tv_nsec = 50000;</div></div><div><br></div><div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>sema_init(&master_sem, 1);</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>ecrt_master_callbacks(</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>master, <span class="Apple-tab-span" style="white-space:pre"> </span>// pointer to master</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>send_callback, <span class="Apple-tab-span" style="white-space:pre"> </span>// fp to send ext callback</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>receive_callback, <span class="Apple-tab-span" style="white-space:pre"> </span>// fp to receive callback</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>master);<span class="Apple-tab-span" style="white-space:pre"> </span>// parameter to callback</div><div><br></div><div><br></div><div><div><span class="Apple-tab-span" style="white-space:pre"> </span>/*</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> * Creates a new process data domain.</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> * For process data exchange, at least one process data domain is needed. </div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> * This method creates a new process data domain and returns a pointer to the new domain object. </div><div><span class="Apple-tab-span" style="white-space:pre"> </span> * This object can be used for registering PDOs and exchanging them in cyclic operation.</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> */</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>domain = ecrt_master_create_domain(master);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>if (!domain)</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>{</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>printk(KERN_INFO PFX "Could not create a domain on the master. Exiting.\n\n");</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>goto out_release_master;<span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>}</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>/*<span class="Apple-tab-span" style="white-space:pre"> </span></div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span> * Register the slaves on the master</div><div> */</div></div><div><br></div><div> ...</div><div> ...</div><div><br></div><div><div> printk(KERN_INFO PFX "Starting cyclic thread.\n");</div>
<div> init_timer(&timer);</div><div> timer.function = cyclic_task;</div><div> timer.expires = jiffies + 10;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>add_timer(&timer);<span class="Apple-tab-span" style="white-space:pre"> </span></div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>printk(KERN_INFO PFX "Durability Service Started.\n");</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>return 0;<span class="Apple-tab-span" style="white-space:pre"> </span></div>
<div><br></div><div>out_release_master:</div><div> printk(KERN_ERR PFX "Releasing master...\n");</div><div> ecrt_release_master(master);</div><div>out_return:</div><div> printk(KERN_ERR PFX "Failed to load. Aborting.\n");</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>return ret;</div><div> } // main</div></div><div><br></div><div><br></div><br><div class="gmail_quote">
On Wed, Aug 15, 2012 at 8:47 PM, Matthieu Bec <span dir="ltr"><<a href="mailto:mbec@gmto.org" target="_blank">mbec@gmto.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
indeed, though the FAQ says<br>
<br>
>Supports any realtime environment through independent architecture.<br>
> RTAI, Xenomai, RT-Preempt, etc.<br>
> Operation possible without any realtime extension at all.<br>
<br>
I am not sure about the second statement or how to make it work. Always had better luck with real-time extensions and never looked back. RT-Preempt is easy to setup and kind of transparent. vanilla kernel was giving me (if I recall correctly) similar semaphores vs. spinlocks issue as you pointed earlier<br>
<br>
Matthieu<br>
<br>
<br>
On 08/15/12 17:53, Thomas Bitsky, Jr. wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm not sure if this is a problem in my source code or a bug in the code<br>
relating to synchronization.<br>
<br>
So, my problem has become this: I can successfully use EoE when the<br>
EtherCAT network is not operational. I can successfully use the EtherCAT<br>
network in Operation if the virtual EoE interface is down, but if I put<br>
the EtherCAT network into Operation and use the callbacks to handle EoE,<br>
the entire computer locks up.<br>
<br>
For reference:<br>
EtherCAT version: stable-1.5<br>
System: Linux laptop14 2.6.32-42-generic-pae #95-Ubuntu SMP Wed Jul 25<br>
16:13:09 UTC 2012 i686 GNU/Linux<br>
I am not using any real-time extensions.<br>
GCC: 4.4<br>
<br>
<br>
The problem could very well be in my source code, although I've matched<br>
it closely to the EtherLAB examples. Once the virtual EoE interface goes<br>
up, the kernel log is filling with the errors I mentioned before:<br>
<br>
[ 2687.384659]<br>
[ 2687.384665] Pid: 0, comm: swapper Tainted: P W<br>
(2.6.32-42-generic-pae #95-Ubuntu) Latitude E6510<br>
[ 2687.384672] EIP: 0060:[<c03ac336>] EFLAGS: 00000202 CPU: 3<br>
[ 2687.384680] EIP is at acpi_idle_enter_bm+0x275/0x2a4<br>
[ 2687.384684] EAX: c088eb4c EBX: 00000ee7 ECX: 00000000 EDX: 03036000<br>
[ 2687.384689] ESI: 00000000 EDI: f6e404cc EBP: f74cbf78 ESP: f74cbf50<br>
[ 2687.384694] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068<br>
[ 2687.384698] CR0: 8005003b CR2: b94c0004 CR3: 00799000 CR4: 000006f0<br>
[ 2687.384703] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000<br>
[ 2687.384707] DR6: ffff0ff0 DR7: 00000400<br>
[ 2687.384710] Call Trace:<br>
[ 2687.384719] [<c04ca54a>] cpuidle_idle_call+0x7a/0x100<br>
[ 2687.384727] [<c01085a4>] cpu_idle+0x94/0xd0<br>
[ 2687.384735] [<c05b31b7>] start_secondary+0xc4/0xc6<br>
[ 2687.393250] BUG: scheduling while atomic: swapper/0/0x10000100<br>
[ 2687.393256] Modules linked in: durability ec_generic ec_e1000<br>
ec_8139too ec_master mii michael_mic arc4 binfmt_misc snd_hda_codec_idt<br>
<br>
<br>
The notable parts of my code are the callback:<br>
<br>
void<br>
send_callback(void *cb_data)<br>
{<br>
ec_master_t *m = (ec_master_t *) cb_data;<br>
down(&master_sem);<br>
ecrt_master_send_ext(m);<br>
up(&master_sem);<br>
}<br>
<br>
void<br>
receive_callback(void *cb_data)<br>
{<br>
ec_master_t *m = (ec_master_t *) cb_data;<br>
down(&master_sem);<br>
ecrt_master_receive(m);<br>
up(&master_sem);<br>
}<br>
If they are not in the program and activated by ecrt_master_callbacks,<br>
then there is no lock-up. Of course, EoE doesn't work. Once I put them<br>
in, the system hangs in about 30 seconds or less. I can't see any<br>
obvious reason for this: it looks like dead lock. I added traces and<br>
watched the kernel log viewer; I think it's locking up in<br>
ecrt_master_send_ext and not returning.<br>
<br>
In any case, I've been working on this for five days. If anyone can<br>
shine some light on what I'm doing wrong, or how I can fix this, I'd<br>
appreciate it.<br>
<br>
Thanks again!<br>
Tom<br>
<br>
<br>
On Wed, Aug 15, 2012 at 1:02 PM, Matthieu Bec <<a href="mailto:mbec@gmto.org" target="_blank">mbec@gmto.org</a><br>
<mailto:<a href="mailto:mbec@gmto.org" target="_blank">mbec@gmto.org</a>>> wrote:<br>
<br>
<br>
# ifconfig eoe0s7 192.168.127.10<br>
# ifconfig eoe0s7 up<br>
<br>
Is there any way to make this happen automatically on startup? In<br>
/etc/network, I created a file called ifcg-eoe0s7 and put in:<br>
<br>
IPADDRESS=<a href="http://192.168.127.10/24" target="_blank">192.168.127.10/24</a> <<a href="http://192.168.127.10/24" target="_blank">http://192.168.127.10/24</a>><br>
<<a href="http://192.168.127.10/24" target="_blank">http://192.168.127.10/24</a>><br>
STARTMODE=auto<br>
<br>
But it must not be working because the virtual interface doesn't<br>
even<br>
have an IP Address until I give it one manually.<br>
<br>
<br>
on fedora:<br>
<br>
# ifup eoe0s7<br>
<br>
best consult the ubuntu forums.<br>
<br>
<br>
<br>
<br>
--<br>
Thomas C. Bitsky Jr.<br>
Lead Developer and Application Engineer<br>
ADC | <a href="http://automateddesign.com" target="_blank">automateddesign.com</a> <<a href="http://automateddesign.com" target="_blank">http://automateddesign.com</a>><br>
P: <a href="tel:630-783-1150" value="+16307831150" target="_blank">630-783-1150</a> <tel:<a href="tel:630-783-1150" value="+16307831150" target="_blank">630-783-1150</a>> F: <a href="tel:630-783-1159" value="+16307831159" target="_blank">630-783-1159</a> <tel:<a href="tel:630-783-1159" value="+16307831159" target="_blank">630-783-1159</a>> M:<br>
<a href="tel:630-632-6679" value="+16306326679" target="_blank">630-632-6679</a> <tel:<a href="tel:630-632-6679" value="+16306326679" target="_blank">630-632-6679</a>><br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
etherlab-dev mailing list<br>
<a href="mailto:etherlab-dev@etherlab.org" target="_blank">etherlab-dev@etherlab.org</a><br>
<a href="http://lists.etherlab.org/mailman/listinfo/etherlab-dev" target="_blank">http://lists.etherlab.org/<u></u>mailman/listinfo/etherlab-dev</a><br>
<br>
</blockquote>
i<span><font color="#888888"><br>
<br>
-- <br>
Matthieu Bec GMTO Corp.<br>
cell: <a href="tel:%2B1%20626%20354%209367" value="+16263549367" target="_blank">+1 626 354 9367</a> P.O. Box 90933<br>
phone: <a href="tel:%2B1%20626%20204%200527" value="+16262040527" target="_blank">+1 626 204 0527</a> Pasadena, CA 91109-0933<br>
<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>Thomas C. Bitsky Jr.<div>Lead Developer and Application Engineer</div><div><font color="#660000">ADC | <a href="http://automateddesign.com" target="_blank">automateddesign.com</a></font></div>
<div>P: <a href="tel:630-783-1150" value="+16307831150" target="_blank">630-783-1150</a> F: <a href="tel:630-783-1159" value="+16307831159" target="_blank">630-783-1159</a> M: <a href="tel:630-632-6679" value="+16306326679" target="_blank">630-632-6679</a></div>
<br>
</div>