[etherlab-users] AL status message 0x0034: "DC Sync Timeout Error" and datagrams TIMED OUT

Sat Jun 23 05:37:17 CEST 2018

Hi,
I'm trying the method from martin 
http://lists.etherlab.org/pipermail/etherlab-users/2018/003304.html
and apply some unofficial patches (dc relative).
"Timed out" error message is still there, but the frequency is lower, may be once per hour.
Is there any other method to fix it at all?

Eric Huang

-----------------------------
>Hi,
>
>I'm trying to use ethercat-1.5.2 communication with Delta A2 servo. I'm using 16.04 Ubuntu with 4.9.47-rt37 kernel, ec_generic with igb driver. I downloaded the code from sourceforge and run the application code development by myself.  I use a loop frequency of 500Hz, and I always get the warning like this:
>
>[11535.480556] EtherCAT WARNING 0: 3 datagrams TIMED OUT!
>[11535.598057] EtherCAT 0: Domain 0: Working counter changed to 18/18
>
>After many times message like this, the master rescan slaves and I'm get the following message:
>
>[13150.763685] EtherCAT 0: Slave states on main device: SAFEOP, OP + ERROR.
>[13150.770129] EtherCAT ERROR 0-0: AL status message 0x0034: "DC Sync Timeout Error".
>[13150.774122] EtherCAT 0-0: Acknowledged state SAFEOP.
>[13151.764521] EtherCAT 0: Domain 0: 4 working counter changes - now 5/18
>
>In fact, I do many homework myself. I'm trying the following methods:
>
>(1)set CPU affinity and disable irqbalance
>
>(2)tune parameters of ethernet card, such as rx/tx-usecs
>
>(3)set the priority of app to 91, and set the priority of Tx/Rx-irq to 92
>
>(4)disable output to console(dmesg -E, I'm not sure it's really work or not)
>
>(5)disable relative option(C-State/Speedstep...) in BIOS
>
>(6)change the net cable
>
>My application is running in userspace, I'm also lock the memory and prevent the page faults,... My realtime loop is as follow:
>
>ecrt_master_receive();
>ecrt_domain_process();
>...
>do a lot of stuff
>...
>ecrt_domain_queue();
>clock_gettime();
>ecrt_master_application_time();
>ecrt_master_sync_reference_clock();
>ecrt_master_sync_slave_clocks();
>ecrt_master_send();
>
>Below is the whole output I'm getting in the kernel log:
>
>EtherCAT: Master driver 1.5.2 2eff7c993a63
>
>[11211.237909] EtherCAT: Requesting master 0...
>[11211.237916] EtherCAT: Successfully requested master 0.
>[11211.238370] EtherCAT 0: Domain0: Logical address 0x00000000, 168 byte, expected working counter 18.
>[11211.238374] EtherCAT 0:   Datagram domain0-0-main: Logical offset 0x00000000, 168 byte, type LRW.
>[11211.238441] EtherCAT 0: Master thread exited.
>[11211.238446] EtherCAT 0: Starting EtherCAT-OP thread.
>[11211.239018] EtherCAT WARNING 0: 1 datagram UNMATCHED!
>[11216.618425] igb 0000:02:00.0 enp2s0: igb: enp2s0 NIC Link is Down
>[11219.616650] igb 0000:02:00.0 enp2s0: igb: enp2s0 NIC Link is Up 100 Mbps Half Duplex, Flow Control: None
>[11226.612199] EtherCAT 0: Domain 0: Working counter changed to 3/18
>[11226.612204] .
>[11227.615032] EtherCAT 0: Domain 0: 4 working counter changes - now 15/18
>[11227.615037] .
>[11232.706757] EtherCAT WARNING 0-5: Slave did not sync after 5001 ms.
>[11232.714811] EtherCAT 0: Domain 0: Working counter changed to 18/18
>[11232.714815] .
>[11232.724444] EtherCAT 0: Slave states on main device: OP.
>[11464.675698] EtherCAT 0: Domain 0: Working counter changed to 0/18
>[11464.675704] .
>[11465.416524] EtherCAT WARNING 0: 3 datagrams TIMED OUT!
>[11465.678088] EtherCAT 0: Domain 0: Working counter changed to 18/18
>[11465.678093] .
>[11534.595654] EtherCAT 0: Domain 0: Working counter changed to 0/18
>[11534.595659] .
>[11535.480556] EtherCAT WARNING 0: 3 datagrams TIMED OUT!
>[11535.598057] EtherCAT 0: Domain 0: Working counter changed to 18/18
>[11535.598062] .
>[11545.455997] EtherCAT 0: Domain 0: Working counter changed to 0/18
>[11545.456002] .
>[11545.489577] EtherCAT WARNING 0: 4 datagrams TIMED OUT!
>[11546.458400] EtherCAT 0: Domain 0: Working counter changed to 18/18
>[11546.458405] .
>[11650.427998] EtherCAT 0: Domain 0: Working counter changed to 0/18
>[11650.428003] .
>[11650.584596] EtherCAT WARNING 0: 2 datagrams TIMED OUT!
>[11651.430394] EtherCAT 0: Domain 0: Working counter changed to 18/18
>[11651.430401] .
>[11809.891785] EtherCAT 0: Domain 0: Working counter changed to 0/18
>[11809.891791] .
>[11810.717649] EtherCAT WARNING 0: 3 datagrams TIMED OUT!
>[11810.894175] EtherCAT 0: Domain 0: Working counter changed to 18/18
>[11810.894179] .
>[11841.651049] NOHZ: local_softirq_pending 80
>[12095.812140] EtherCAT 0: Domain 0: Working counter changed to 0/18
>[12095.812145] .
>[12095.951751] EtherCAT WARNING 0: 3 datagrams TIMED OUT!
>[12096.814541] EtherCAT 0: Domain 0: Working counter changed to 18/18
>[12096.814546] .
>[12291.250098] usb 1-1.1.3: USB disconnect, device number 5
>[12292.965286] usb 1-1.1: USB disconnect, device number 3
>[12311.063783] usb 1-1.1: new low-speed USB device number 7 using ehci-pci
>[12311.150559] usb 1-1.1: New USB device found, idVendor=413c, idProduct=2107
>[12311.150565] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
>[12311.150568] usb 1-1.1: Product: Dell USB Entry Keyboard
>[12311.150575] usb 1-1.1: Manufacturer: Dell
>[12311.155244] input: Dell Dell USB Entry Keyboard as /devices/pci0000:00/0000:00:1d.0/usb1/1-1/1-1.1/1-1.1:1.0/0003:413C:2107.0003/input/input7
>[12311.207877] hid-generic 0003:413C:2107.0003: input,hidraw1: USB HID v1.10 Keyboard [Dell Dell USB Entry Keyboard] on usb-0000:00:1d.0-1.1/input0
>[13148.651274] EtherCAT 0: Domain 0: Working counter changed to 0/18
>[13148.651279] .
>[13148.652828] EtherCAT 0: 0 slave(s) responding on main device.
>[13148.656820] EtherCAT 0: 6 slave(s) responding on main device.
>[13148.656825] EtherCAT 0: Slave states on main device: OP.
>[13148.661277] EtherCAT 0: Scanning bus.
>[13149.511156] EtherCAT WARNING 0: 3 datagrams TIMED OUT!
>[13149.653676] EtherCAT 0: Domain 0: Working counter changed to 18/18
>[13149.653681] .
>[13150.734111] EtherCAT 0: Bus scanning completed in 2076 ms.
>[13150.734116] EtherCAT 0: Using slave 0 as DC reference clock.
>[13150.762120] EtherCAT 0: Domain 0: Working counter changed to 16/18
>[13150.762125] .
>[13150.763685] EtherCAT 0: Slave states on main device: SAFEOP, OP + ERROR.
>[13150.770129] EtherCAT ERROR 0-0: AL status message 0x0034: "DC Sync Timeout Error".
>[13150.774122] EtherCAT 0-0: Acknowledged state SAFEOP.
>[13151.764521] EtherCAT 0: Domain 0: 4 working counter changes - now 5/18
>[13151.764526] .
>[13155.928201] EtherCAT WARNING 0-0: Slave did not sync after 5000 ms.
>[13155.936188] EtherCAT 0: Domain 0: Working counter changed to 8/18
>[13155.936193] .
>[13155.946192] EtherCAT ERROR 0-1: AL status message 0x0034: "DC Sync Timeout Error".
>[13155.950191] EtherCAT 0-1: Acknowledged state SAFEOP.
>[13156.938602] EtherCAT 0: Domain 0: 2 working counter changes - now 7/18
>[13156.938607] .
>[13159.755727] EtherCAT 0: Domain 0: Working counter changed to 10/18
>[13159.755732] .
>[13159.765730] EtherCAT ERROR 0-2: AL status message 0x0034: "DC Sync Timeout Error".
>[13159.769719] EtherCAT 0-2: Acknowledged state SAFEOP.
>[13160.758119] EtherCAT 0: Domain 0: 2 working counter changes - now 9/18
>[13160.758123] .
>[13164.379567] EtherCAT 0: Domain 0: Working counter changed to 12/18
>[13164.379571] .
>[13164.391579] EtherCAT ERROR 0-3: AL status message 0x0034: "DC Sync Timeout Error".
>[13164.395569] EtherCAT 0-3: Acknowledged state SAFEOP.
>[13165.381977] EtherCAT 0: Domain 0: 2 working counter changes - now 11/18
>[13165.381982] .
>[13169.549633] EtherCAT WARNING 0-3: Slave did not sync after 5000 ms.
>[13169.557638] EtherCAT 0: Domain 0: Working counter changed to 14/18
>[13169.557643] .
>[13169.567642] EtherCAT ERROR 0-4: AL status message 0x0034: "DC Sync Timeout Error".
>[13169.571654] EtherCAT 0-4: Acknowledged state SAFEOP.
>[13170.560055] EtherCAT 0: Domain 0: 2 working counter changes - now 13/18
>[13170.560061] .
>[13174.725708] EtherCAT WARNING 0-4: Slave did not sync after 5000 ms.
>[13174.733715] EtherCAT 0: Domain 0: Working counter changed to 16/18
>[13174.733721] .
>[13174.745738] EtherCAT ERROR 0-5: AL status message 0x0034: "DC Sync Timeout Error".
>[13174.749715] EtherCAT 0-5: Acknowledged state SAFEOP.
>[13175.393536] EtherCAT 0: Slave states on main device: OP.
>[13175.736114] EtherCAT 0: Domain 0: 3 working counter changes - now 18/18
>[13175.736122] .
>[13178.967523] EtherCAT 0: Master thread exited.
>[13178.967562] EtherCAT 0: Starting EtherCAT-IDLE thread.
>[13178.967680] EtherCAT ERROR 0-3: Failed to receive AL state datagram:
>[13178.967684] Datagram
>[13178.967687] initialized
>[13178.967689] .
>[13178.967744] EtherCAT 0: Releasing master...
>[13178.967749] EtherCAT 0: Released.
>[13178.989988] EtherCAT 0: Slave states on main device: PREOP.
>[13179.454436] EtherCAT WARNING: Datagram ffff88005e6e2678 (master-fsm) was SKIPPED 1 time.
>
>What am I doing wrong? Is there anyone please offer me some advice, to fix them?
>
>Regards,
>
>Eirc Huang