[etherlab-users] question about hang on EtherLAB EtherCAT master stack and ethernet hang

[Bruno Bittner]

We are using a TI am3359 processor running the Arago Linux dsitribution 
from TI which is based on Linux 3.2.  We are using RT_PREEMPT patch for 
3.20 rt10.  I built EtherLAB EtherCAT master version 1.5.0 
(etherlabmaster-4adb56c4198d).  It works great most of the time. 
Occasionaly we get a hang.  I enabled the debug msg output in the driver. 
We suspect this is a driver issued, but I was asked to post here in case 
anyone may have seen this before and/or can provide clues.

Here is the output at the point of hang (the ethercat master continues to 
run, but based on the ethernet activity lights, only a few packets get out 
every few seconds):

[21122.115318] net_ratelimit: 12515 callbacks suppressed
[21127.125317] net_ratelimit: 12515 callbacks suppressed
[21132.135317] net_ratelimit: 12515 callbacks suppressed
[21137.145319] net_ratelimit: 12515 callbacks suppressed
[21138.516040] EtherCAT 0: Domain 0: Working counter changed to 0/27.
[21138.635127] EtherCAT ERROR 0-1: Failed to receive AL state datagram: 
Datagram timed out.
[21138.995123] EtherCAT WARNING 0: 39 datagrams TIMED OUT!
[21139.995138] EtherCAT WARNING 0: 83 datagrams TIMED OUT!
[21140.995136] EtherCAT WARNING 0: 84 datagrams TIMED OUT!
[21141.995139] EtherCAT WARNING 0: 84 datagrams TIMED OUT!
[21142.295786] net_ratelimit: 3417 callbacks suppressed
[21142.295806] net eth0: desc submit failed
[21142.995138] EtherCAT WARNING 0: 84 datagrams TIMED OUT!
[21143.995133] EtherCAT WARNING 0: 68 datagrams TIMED OUT!
[21144.295770] net eth0: desc submit failed
[21146.295837] net eth0: desc submit failed
[21147.015127] ------------[ cut here ]------------
[21147.015192] WARNING: at net/sched/sch_generic.c:255 
dev_watchdog+0x28c/0x29c()
[21147.015206] NETDEV WATCHDOG: eth0 (cpsw): transmit queue 0 timed out
[21147.015215] Modules linked in: g_ether ec_generic(O) ec_master(O)
[21147.015236] Backtrace: 
[21147.015270] [<c0017b64>] (dump_backtrace+0x0/0x110) from [<c03c2e84>] 
(dump_stack+0x18/0x1c)
[21147.015283]  r6:c04d0b64 r5:000000ff r4:cf82fe70 r3:00000000
[21147.015316] [<c03c2e6c>] (dump_stack+0x0/0x1c) from [<c003f62c>] 
(warn_slowpath_common+0x5c/0x6c)
[21147.015336] [<c003f5d0>] (warn_slowpath_common+0x0/0x6c) from 
[<c003f6e0>] (warn_slowpath_fmt+0x38/0x40)
[21147.015348]  r8:c05bad78 r7:c0560a58 r6:00000000 r5:cfae49d8 
r4:cfae4800
[21147.015366] r3:00000009
[21147.015384] [<c003f6a8>] (warn_slowpath_fmt+0x0/0x40) from [<c032b140>] 
(dev_watchdog+0x28c/0x29c)
[21147.015395]  r3:cfae4800 r2:c04d0b7c
[21147.015424] [<c032aeb4>] (dev_watchdog+0x0/0x29c) from [<c004aaa8>] 
(run_timer_softirq+0x110/0x23c)
[21147.015449] [<c004a998>] (run_timer_softirq+0x0/0x23c) from 
[<c0044f80>] (__do_softirq_common+0xd8/0x174)
[21147.015471] [<c0044ea8>] (__do_softirq_common+0x0/0x174) from 
[<c00450dc>] (__thread_do_softirq+0xc0/0x10c)
[21147.015492] [<c004501c>] (__thread_do_softirq+0x0/0x10c) from 
[<c00451ac>] (run_ksoftirqd+0x84/0x168)
[21147.015504]  r6:cf82ff84 r5:cf82e000 r4:00000000 r3:00000002
[21147.015540] [<c0045128>] (run_ksoftirqd+0x0/0x168) from [<c0059cf0>] 
(kthread+0x90/0x94)
[21147.015550]  r8:00000000 r7:00000013 r6:c0045128 r5:00000000 
r4:cf82bef4
[21147.015580] [<c0059c60>] (kthread+0x0/0x94) from [<c0042b7c>] 
(do_exit+0x0/0x6b8)
[21147.015590]  r6:c0042b7c r5:c0059c60 r4:cf82bef4
[21147.015606] ---[ end trace 0000000000000002 ]---
[21147.015625] net eth0: transmit timeout, restarting dma
[21147.018375] net eth0: desc submit failed
[21147.995137] EtherCAT WARNING 0: 80 datagrams TIMED OUT!
[21148.295807] net eth0: desc submit failed
[21148.995133] EtherCAT WARNING 0: 79 datagrams TIMED OUT!
[21150.295807] net eth0: desc submit failed
[21152.295790] net eth0: desc submit failed
[21154.295778] net eth0: desc submit failed
[21156.295827] net eth0: desc submit failed
[21157.015143] net eth0: transmit timeout, restarting dma
[21157.017926] net eth0: desc submit failed
[21157.995139] EtherCAT WARNING 0: 81 datagrams TIMED OUT!
[21158.295853] net eth0: desc submit failed
[21158.995134] EtherCAT WARNING 0: 79 datagrams TIMED OUT!
[21160.295797] net eth0: desc submit failed
[21162.295780] net eth0: desc submit failed
[21164.295834] net eth0: desc submit failed
[21166.295819] net eth0: desc submit failed
[21167.015141] net eth0: transmit timeout, restarting dma
[21167.017909] net eth0: desc submit failed
[21167.995140] EtherCAT WARNING 0: 81 datagrams TIMED OUT!
[21168.295842] net eth0: desc submit failed
[21168.995135] EtherCAT WARNING 0: 76 datagrams TIMED OUT!
[21170.295786] net eth0: desc submit failed

Regards,

Bruno Bittner
Product Development Engineer - Software
---------------------------------------------------
SICK, Inc.
---------------------------------------------------
6900 West 110th Street
Bloomington, MN 55438 USA
Phone: 952.941.6780
Fax: 952.941.9287
Direct: 952.829.4814
http://www.sickusa.com