[etherlab-users] Sync problems and DC mode

Mon Jan 22 23:24:20 CET 2018

Hi everyone,

I've a machine with 25 Hiwin servos working usign linuxcnc and Sittner HAL
etherCAT component <https://github.com/sittner/linuxcnc-ethercat>

My problem is I am suffering some random desynchronizations. Dmesg output:
(full text as attachment dmesgoutput.txt)

Jan 22 12:01:16 kernel: [11788.586944] EtherCAT 0: Domain 0: 6 working
counter changes - now 75/75.

at 12:01:16 the machine is working ok and everything is well but suddenly
(most of times without doing anything on the application)

Jan 22 13:04:31 kernel: [15578.066190] EtherCAT 0: Domain 0: Working
counter changed to 27/75.
Jan 22 13:04:31 kernel: [15578.074179] EtherCAT 0: 9 slave(s) responding on
main device.
Jan 22 13:04:31 kernel: [15578.074181] EtherCAT 0: Slave states on main
device: OP.
Jan 22 13:04:31 kernel: [15578.077148] EtherCAT 0: Scanning bus.

and so on... until it gets to sync everything again.

I've reading a lot of the etherlab users mailing list and I've found
something I think could help me with the problem.
The application that I'm running (what most of people of linuxcnc run) is
using the first method that Graeme Foot describes here
<http://lists.etherlab.org/pipermail/etherlab-users/2016/003013.html>
In the file lcec_main.c
<https://github.com/sittner/linuxcnc-ethercat/blob/master/src/lcec_main.c>
from line 1119 you can see how he makes it.

I've tried to modify it in many ways but I have not been able to make it
works propertly.
With this version
<https://github.com/narogon/linuxcnc-ethercat/commit/e4ab86ba6167ced532e49904059df580062b2d97#diff-059a684a933530837771b5a249433ff3>
(also as attachment lcec_main.c) I get the servos sync and OP but it seems
that the PDO doesn't arrive for some of the slaves (no idea why).

Could you help me with this problem? Could be the DC method the origin of
the  desynchronizations or must be something different? How could I get the
application working with the 2nd mode that Foot described?

Many thanks,

Best Regards
Nacho Rosales
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20180122/be12ed46/attachment-0002.htm>
-------------- next part --------------
Jan 22 12:01:16 kernel: [11788.586944] EtherCAT 0: Domain 0: 6 working counter changes - now 75/75.
Jan 22 12:10:17 kernel: [12329.376472] [Hardware Error]: Machine check events logged
Jan 22 12:22:32 kernel: [13063.594366] [Hardware Error]: Machine check events logged
Jan 22 12:36:27 kernel: [13896.896685] [Hardware Error]: Machine check events logged
Jan 22 12:50:51 kernel: [14759.573840] [Hardware Error]: Machine check events logged
Jan 22 13:04:31 kernel: [15578.066190] EtherCAT 0: Domain 0: Working counter changed to 27/75.
Jan 22 13:04:31 kernel: [15578.074179] EtherCAT 0: 9 slave(s) responding on main device.
Jan 22 13:04:31 kernel: [15578.074181] EtherCAT 0: Slave states on main device: OP.
Jan 22 13:04:31 kernel: [15578.077148] EtherCAT 0: Scanning bus.
Jan 22 13:04:32 kernel: [15579.680451] EtherCAT 0: Domain 0: Working counter changed to 28/75.
Jan 22 13:04:33 kernel: [15580.274450] EtherCAT WARNING: Datagram f69602cc (domain0-0-main) was SKIPPED 1 time.
Jan 22 13:04:33 kernel: [15580.621858] EtherCAT WARNING 0: 1 datagram TIMED OUT!
Jan 22 13:04:33 kernel: [15580.681762] EtherCAT 0: Domain 0: 2 working counter changes - now 42/75.
Jan 22 13:04:33 kernel: [15580.955298] EtherCAT 0: Bus scanning completed in 2884 ms.
Jan 22 13:04:33 kernel: [15580.955300] EtherCAT 0: Using slave 0 as DC reference clock.
Jan 22 13:04:33 kernel: [15580.955302] EtherCAT ERROR 0: Failed to calculate bus topology.
Jan 22 13:04:33 kernel: [15580.977276] EtherCAT 0: 37 slave(s) responding on main device.
Jan 22 13:04:33 kernel: [15580.977277] EtherCAT 0: Slave states on main device: INIT, PREOP, SAFEOP, OP + ERROR.
Jan 22 13:04:33 kernel: [15580.980253] EtherCAT 0: Scanning bus.
Jan 22 13:04:36 kernel: [15583.850415] EtherCAT WARNING 0-9: Slave has state error bit set (SAFEOP + ERROR)!
Jan 22 13:04:37 kernel: [15584.157896] EtherCAT WARNING 0-10: Slave has state error bit set (INIT + ERROR)!
Jan 22 13:04:37 kernel: [15584.475361] EtherCAT WARNING 0-11: Slave has state error bit set (SAFEOP + ERROR)!
Jan 22 13:04:37 kernel: [15584.785838] EtherCAT WARNING 0-12: Slave has state error bit set (SAFEOP + ERROR)!
Jan 22 13:04:38 kernel: [15585.106297] EtherCAT WARNING 0-13: Slave has state error bit set (SAFEOP + ERROR)!
Jan 22 13:04:38 kernel: [15585.425758] EtherCAT WARNING 0-14: Slave has state error bit set (SAFEOP + ERROR)!
Jan 22 13:04:38 kernel: [15585.746218] EtherCAT WARNING 0-15: Slave has state error bit set (SAFEOP + ERROR)!
Jan 22 13:04:39 kernel: [15586.067676] EtherCAT WARNING 0-16: Slave has state error bit set (SAFEOP + ERROR)!
Jan 22 13:04:39 kernel: [15586.376157] EtherCAT WARNING 0-17: Slave has state error bit set (SAFEOP + ERROR)!
Jan 22 13:04:39 kernel: [15586.683638] EtherCAT WARNING 0-18: Slave has state error bit set (SAFEOP + ERROR)!
Jan 22 13:04:39 kernel: [15586.994114] EtherCAT WARNING 0-19: Slave has state error bit set (SAFEOP + ERROR)!
Jan 22 13:04:40 kernel: [15587.315572] EtherCAT WARNING 0-20: Slave has state error bit set (SAFEOP + ERROR)!
Jan 22 13:04:40 kernel: [15587.636033] EtherCAT WARNING 0-21: Slave has state error bit set (SAFEOP + ERROR)!
Jan 22 13:04:40 kernel: [15587.944511] EtherCAT WARNING 0-22: Slave has state error bit set (SAFEOP + ERROR)!
Jan 22 13:04:41 kernel: [15588.253990] EtherCAT WARNING 0-23: Slave has state error bit set (SAFEOP + ERROR)!
Jan 22 13:04:41 kernel: [15588.574449] EtherCAT WARNING 0-24: Slave has state error bit set (SAFEOP + ERROR)!
Jan 22 13:04:45 kernel: [15592.503824] EtherCAT 0: Bus scanning completed in 11540 ms.
Jan 22 13:04:45 kernel: [15592.503827] EtherCAT 0: Using slave 0 as DC reference clock.
Jan 22 13:04:45 kernel: [15592.580710] EtherCAT 0: Slave states on main device: PREOP, SAFEOP, OP + ERROR.
Jan 22 13:04:45 kernel: [15592.621625] EtherCAT ERROR 0-9: AL status message 0x001B: "Sync manager watchdog".
Jan 22 13:04:45 kernel: [15592.624622] EtherCAT 0-9: Acknowledged state SAFEOP.
Jan 22 13:04:45 kernel: [15592.624626] EtherCAT 0: Domain 0: Working counter changed to 44/75.
Jan 22 13:04:46 kernel: [15593.565035] EtherCAT ERROR 0-11: AL status message 0x001B: "Sync manager watchdog".
Jan 22 13:04:46 kernel: [15593.568032] EtherCAT 0-11: Acknowledged state SAFEOP.
Jan 22 13:04:46 kernel: [15593.627934] EtherCAT 0: Domain 0: 6 working counter changes - now 45/75.
Jan 22 13:04:46 kernel: [15593.680840] EtherCAT ERROR 0-12: AL status message 0x001B: "Sync manager watchdog".
Jan 22 13:04:46 kernel: [15593.683837] EtherCAT 0-12: Acknowledged state SAFEOP.
Jan 22 13:04:46 kernel: [15593.885497] EtherCAT ERROR 0-13: AL status message 0x001B: "Sync manager watchdog".
Jan 22 13:04:46 kernel: [15593.889490] EtherCAT 0-13: Acknowledged state SAFEOP.
Jan 22 13:04:47 kernel: [15594.105128] EtherCAT ERROR 0-14: AL status message 0x001B: "Sync manager watchdog".
Jan 22 13:04:47 kernel: [15594.109120] EtherCAT 0-14: Acknowledged state SAFEOP.
Jan 22 13:04:47 kernel: [15594.324757] EtherCAT ERROR 0-15: AL status message 0x001B: "Sync manager watchdog".
Jan 22 13:04:47 kernel: [15594.328750] EtherCAT 0-15: Acknowledged state SAFEOP.
Jan 22 13:04:47 kernel: [15594.544387] EtherCAT ERROR 0-16: AL status message 0x001B: "Sync manager watchdog".
Jan 22 13:04:47 kernel: [15594.548379] EtherCAT 0-16: Acknowledged state SAFEOP.
Jan 22 13:04:47 kernel: [15594.631241] EtherCAT 0: Domain 0: 21 working counter changes - now 56/75.
Jan 22 13:04:47 kernel: [15594.764017] EtherCAT ERROR 0-17: AL status message 0x001B: "Sync manager watchdog".
Jan 22 13:04:47 kernel: [15594.768009] EtherCAT 0-17: Acknowledged state SAFEOP.
Jan 22 13:04:47 kernel: [15594.983647] EtherCAT ERROR 0-18: AL status message 0x001B: "Sync manager watchdog".
Jan 22 13:04:47 kernel: [15594.987639] EtherCAT 0-18: Acknowledged state SAFEOP.
Jan 22 13:04:48 kernel: [15595.203276] EtherCAT ERROR 0-19: AL status message 0x001B: "Sync manager watchdog".
Jan 22 13:04:48 kernel: [15595.207269] EtherCAT 0-19: Acknowledged state SAFEOP.
Jan 22 13:04:48 kernel: [15595.422915] EtherCAT ERROR 0-20: AL status message 0x001B: "Sync manager watchdog".
Jan 22 13:04:48 kernel: [15595.426911] EtherCAT 0-20: Acknowledged state SAFEOP.
Jan 22 13:04:48 kernel: [15595.633551] EtherCAT 0: Domain 0: 17 working counter changes - now 66/75.
Jan 22 13:04:48 kernel: [15595.642535] EtherCAT ERROR 0-21: AL status message 0x001B: "Sync manager watchdog".
Jan 22 13:04:48 kernel: [15595.646528] EtherCAT 0-21: Acknowledged state SAFEOP.
Jan 22 13:04:48 kernel: [15595.862166] EtherCAT ERROR 0-22: AL status message 0x001B: "Sync manager watchdog".
Jan 22 13:04:48 kernel: [15595.866158] EtherCAT 0-22: Acknowledged state SAFEOP.
Jan 22 13:04:49 kernel: [15596.081795] EtherCAT ERROR 0-23: AL status message 0x001B: "Sync manager watchdog".
Jan 22 13:04:49 kernel: [15596.085788] EtherCAT 0-23: Acknowledged state SAFEOP.
Jan 22 13:04:49 kernel: [15596.301425] EtherCAT ERROR 0-24: AL status message 0x001B: "Sync manager watchdog".
Jan 22 13:04:49 kernel: [15596.305418] EtherCAT 0-24: Acknowledged state SAFEOP.
Jan 22 13:04:49 kernel: [15596.572968] EtherCAT ERROR 0-0: AL status message 0x001A: "Synchronization error".
Jan 22 13:04:49 kernel: [15596.576959] EtherCAT 0-0: Acknowledged state SAFEOP.
Jan 22 13:04:49 kernel: [15596.635861] EtherCAT 0: Domain 0: 18 working counter changes - now 72/75.
Jan 22 13:04:49 kernel: [15596.796591] EtherCAT ERROR 0-2: AL status message 0x001A: "Synchronization error".
Jan 22 13:04:49 kernel: [15596.800583] EtherCAT 0-2: Acknowledged state SAFEOP.
Jan 22 13:04:49 kernel: [15597.020213] EtherCAT ERROR 0-4: AL status message 0x001A: "Synchronization error".
Jan 22 13:04:49 kernel: [15597.024206] EtherCAT 0-4: Acknowledged state SAFEOP.
Jan 22 13:04:50 kernel: [15597.244834] EtherCAT ERROR 0-6: AL status message 0x001A: "Synchronization error".
Jan 22 13:04:50 kernel: [15597.248827] EtherCAT 0-6: Acknowledged state SAFEOP.
Jan 22 13:04:50 kernel: [15597.468458] EtherCAT ERROR 0-8: AL status message 0x001A: "Synchronization error".
Jan 22 13:04:50 kernel: [15597.472450] EtherCAT 0-8: Acknowledged state SAFEOP.
Jan 22 13:04:50 kernel: [15597.639169] EtherCAT 0: Domain 0: 12 working counter changes - now 72/75.
Jan 22 13:04:50 kernel: [15597.797923] EtherCAT 0: Slave states on main device: PREOP, OP.
Jan 22 13:04:51 kernel: [15598.640501] EtherCAT 0: Domain 0: 2 working counter changes - now 75/75.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lcec_main.c
Type: text/x-csrc
Size: 45304 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20180122/be12ed46/attachment-0003.c>