[etherlab-users] AL status message 0x0034: "DC Sync Timeout Error" and datagrams TIMED OUT

Mon Jun 11 11:33:20 CEST 2018

Hi,

I'm trying to use ethercat-1.5.2 communication with Delta A2 servo. I'm using 16.04 Ubuntu with 4.9.47-rt37 kernel, ec_generic with igb driver. I downloaded the code from sourceforge and run the application code development by myself.  I use a loop frequency of 500Hz, and I always get the warning like this:

[11535.480556] EtherCAT WARNING 0: 3 datagrams TIMED OUT!
[11535.598057] EtherCAT 0: Domain 0: Working counter changed to 18/18

After many times message like this, the master rescan slaves and I'm get the following message:

[13150.763685] EtherCAT 0: Slave states on main device: SAFEOP, OP + ERROR.
[13150.770129] EtherCAT ERROR 0-0: AL status message 0x0034: "DC Sync Timeout Error".
[13150.774122] EtherCAT 0-0: Acknowledged state SAFEOP.
[13151.764521] EtherCAT 0: Domain 0: 4 working counter changes - now 5/18

In fact, I do many homework myself. I'm trying the following methods:

(1)set CPU affinity and disable irqbalance

(2)tune parameters of ethernet card, such as rx/tx-usecs

(3)set the priority of app to 91, and set the priority of Tx/Rx-irq to 92

(4)disable output to console(dmesg -E, I'm not sure it's really work or not)

(5)disable relative option(C-State/Speedstep...) in BIOS

(6)change the net cable

My application is running in userspace, I'm also lock the memory and prevent the page faults,... My realtime loop is as follow:

ecrt_master_receive();
ecrt_domain_process();
...
do a lot of stuff
...
ecrt_domain_queue();
clock_gettime();
ecrt_master_application_time();
ecrt_master_sync_reference_clock();
ecrt_master_sync_slave_clocks();
ecrt_master_send();

Below is the whole output I'm getting in the kernel log:

EtherCAT: Master driver 1.5.2 2eff7c993a63

[11211.237909] EtherCAT: Requesting master 0...
[11211.237916] EtherCAT: Successfully requested master 0.
[11211.238370] EtherCAT 0: Domain0: Logical address 0x00000000, 168 byte, expected working counter 18.
[11211.238374] EtherCAT 0:   Datagram domain0-0-main: Logical offset 0x00000000, 168 byte, type LRW.
[11211.238441] EtherCAT 0: Master thread exited.
[11211.238446] EtherCAT 0: Starting EtherCAT-OP thread.
[11211.239018] EtherCAT WARNING 0: 1 datagram UNMATCHED!
[11216.618425] igb 0000:02:00.0 enp2s0: igb: enp2s0 NIC Link is Down
[11219.616650] igb 0000:02:00.0 enp2s0: igb: enp2s0 NIC Link is Up 100 Mbps Half Duplex, Flow Control: None
[11226.612199] EtherCAT 0: Domain 0: Working counter changed to 3/18
[11226.612204] .
[11227.615032] EtherCAT 0: Domain 0: 4 working counter changes - now 15/18
[11227.615037] .
[11232.706757] EtherCAT WARNING 0-5: Slave did not sync after 5001 ms.
[11232.714811] EtherCAT 0: Domain 0: Working counter changed to 18/18
[11232.714815] .
[11232.724444] EtherCAT 0: Slave states on main device: OP.
[11464.675698] EtherCAT 0: Domain 0: Working counter changed to 0/18
[11464.675704] .
[11465.416524] EtherCAT WARNING 0: 3 datagrams TIMED OUT!
[11465.678088] EtherCAT 0: Domain 0: Working counter changed to 18/18
[11465.678093] .
[11534.595654] EtherCAT 0: Domain 0: Working counter changed to 0/18
[11534.595659] .
[11535.480556] EtherCAT WARNING 0: 3 datagrams TIMED OUT!
[11535.598057] EtherCAT 0: Domain 0: Working counter changed to 18/18
[11535.598062] .
[11545.455997] EtherCAT 0: Domain 0: Working counter changed to 0/18
[11545.456002] .
[11545.489577] EtherCAT WARNING 0: 4 datagrams TIMED OUT!
[11546.458400] EtherCAT 0: Domain 0: Working counter changed to 18/18
[11546.458405] .
[11650.427998] EtherCAT 0: Domain 0: Working counter changed to 0/18
[11650.428003] .
[11650.584596] EtherCAT WARNING 0: 2 datagrams TIMED OUT!
[11651.430394] EtherCAT 0: Domain 0: Working counter changed to 18/18
[11651.430401] .
[11809.891785] EtherCAT 0: Domain 0: Working counter changed to 0/18
[11809.891791] .
[11810.717649] EtherCAT WARNING 0: 3 datagrams TIMED OUT!
[11810.894175] EtherCAT 0: Domain 0: Working counter changed to 18/18
[11810.894179] .
[11841.651049] NOHZ: local_softirq_pending 80
[12095.812140] EtherCAT 0: Domain 0: Working counter changed to 0/18
[12095.812145] .
[12095.951751] EtherCAT WARNING 0: 3 datagrams TIMED OUT!
[12096.814541] EtherCAT 0: Domain 0: Working counter changed to 18/18
[12096.814546] .
[12291.250098] usb 1-1.1.3: USB disconnect, device number 5
[12292.965286] usb 1-1.1: USB disconnect, device number 3
[12311.063783] usb 1-1.1: new low-speed USB device number 7 using ehci-pci
[12311.150559] usb 1-1.1: New USB device found, idVendor=413c, idProduct=2107
[12311.150565] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[12311.150568] usb 1-1.1: Product: Dell USB Entry Keyboard
[12311.150575] usb 1-1.1: Manufacturer: Dell
[12311.155244] input: Dell Dell USB Entry Keyboard as /devices/pci0000:00/0000:00:1d.0/usb1/1-1/1-1.1/1-1.1:1.0/0003:413C:2107.0003/input/input7
[12311.207877] hid-generic 0003:413C:2107.0003: input,hidraw1: USB HID v1.10 Keyboard [Dell Dell USB Entry Keyboard] on usb-0000:00:1d.0-1.1/input0
[13148.651274] EtherCAT 0: Domain 0: Working counter changed to 0/18
[13148.651279] .
[13148.652828] EtherCAT 0: 0 slave(s) responding on main device.
[13148.656820] EtherCAT 0: 6 slave(s) responding on main device.
[13148.656825] EtherCAT 0: Slave states on main device: OP.
[13148.661277] EtherCAT 0: Scanning bus.
[13149.511156] EtherCAT WARNING 0: 3 datagrams TIMED OUT!
[13149.653676] EtherCAT 0: Domain 0: Working counter changed to 18/18
[13149.653681] .
[13150.734111] EtherCAT 0: Bus scanning completed in 2076 ms.
[13150.734116] EtherCAT 0: Using slave 0 as DC reference clock.
[13150.762120] EtherCAT 0: Domain 0: Working counter changed to 16/18
[13150.762125] .
[13150.763685] EtherCAT 0: Slave states on main device: SAFEOP, OP + ERROR.
[13150.770129] EtherCAT ERROR 0-0: AL status message 0x0034: "DC Sync Timeout Error".
[13150.774122] EtherCAT 0-0: Acknowledged state SAFEOP.
[13151.764521] EtherCAT 0: Domain 0: 4 working counter changes - now 5/18
[13151.764526] .
[13155.928201] EtherCAT WARNING 0-0: Slave did not sync after 5000 ms.
[13155.936188] EtherCAT 0: Domain 0: Working counter changed to 8/18
[13155.936193] .
[13155.946192] EtherCAT ERROR 0-1: AL status message 0x0034: "DC Sync Timeout Error".
[13155.950191] EtherCAT 0-1: Acknowledged state SAFEOP.
[13156.938602] EtherCAT 0: Domain 0: 2 working counter changes - now 7/18
[13156.938607] .
[13159.755727] EtherCAT 0: Domain 0: Working counter changed to 10/18
[13159.755732] .
[13159.765730] EtherCAT ERROR 0-2: AL status message 0x0034: "DC Sync Timeout Error".
[13159.769719] EtherCAT 0-2: Acknowledged state SAFEOP.
[13160.758119] EtherCAT 0: Domain 0: 2 working counter changes - now 9/18
[13160.758123] .
[13164.379567] EtherCAT 0: Domain 0: Working counter changed to 12/18
[13164.379571] .
[13164.391579] EtherCAT ERROR 0-3: AL status message 0x0034: "DC Sync Timeout Error".
[13164.395569] EtherCAT 0-3: Acknowledged state SAFEOP.
[13165.381977] EtherCAT 0: Domain 0: 2 working counter changes - now 11/18
[13165.381982] .
[13169.549633] EtherCAT WARNING 0-3: Slave did not sync after 5000 ms.
[13169.557638] EtherCAT 0: Domain 0: Working counter changed to 14/18
[13169.557643] .
[13169.567642] EtherCAT ERROR 0-4: AL status message 0x0034: "DC Sync Timeout Error".
[13169.571654] EtherCAT 0-4: Acknowledged state SAFEOP.
[13170.560055] EtherCAT 0: Domain 0: 2 working counter changes - now 13/18
[13170.560061] .
[13174.725708] EtherCAT WARNING 0-4: Slave did not sync after 5000 ms.
[13174.733715] EtherCAT 0: Domain 0: Working counter changed to 16/18
[13174.733721] .
[13174.745738] EtherCAT ERROR 0-5: AL status message 0x0034: "DC Sync Timeout Error".
[13174.749715] EtherCAT 0-5: Acknowledged state SAFEOP.
[13175.393536] EtherCAT 0: Slave states on main device: OP.
[13175.736114] EtherCAT 0: Domain 0: 3 working counter changes - now 18/18
[13175.736122] .
[13178.967523] EtherCAT 0: Master thread exited.
[13178.967562] EtherCAT 0: Starting EtherCAT-IDLE thread.
[13178.967680] EtherCAT ERROR 0-3: Failed to receive AL state datagram:
[13178.967684] Datagram
[13178.967687] initialized
[13178.967689] .
[13178.967744] EtherCAT 0: Releasing master...
[13178.967749] EtherCAT 0: Released.
[13178.989988] EtherCAT 0: Slave states on main device: PREOP.
[13179.454436] EtherCAT WARNING: Datagram ffff88005e6e2678 (master-fsm) was SKIPPED 1 time.

What am I doing wrong? Is there anyone please offer me some advice, to fix them?

Regards,

Eirc Huang