[etherlab-users] DC drift issue

Wed Jan 8 23:58:01 CET 2020

You should probably start with instrumentation -- try adding performance counters around your cyclic loop and verify that you're not getting any timing blowouts (post them to another thread to actually process/log, though).

You can also enable the debug interface or insert a switch with another PC between the master and first slave, and then use Wireshark to log your packets and verify the timing of the LWR datagrams in particular.

Be careful with things like slots/signals -- depending on particular implementation, they can execute on the same thread or a separate thread, and often use mutexes in their implementations.  Mutexes are very dangerous to the performance of RT cyclic loops, especially when their "other end" is a non-RT thread.  Prefer using lock-free queues (eg. a simple circular buffer) to communicate between RT and non-RT threads.

Also, using CLOCK_REALTIME is not recommended -- this is the wall-clock time, which is affected by things like NTP resyncing and OS suspend.  Prefer using CLOCK_MONOTONIC instead.  (Some people recommend CLOCK_MONOTONIC_RAW, as this is also immune to NTP drift compensation, but so far at least I think it's better to keep that so that you're less subject to the whims of your clock hardware.  And the raw clock is slower as well, counterintuitively.)

Gavin Lambert
Senior Software Developer

COMPAC SORTING EQUIPMENT LTD | 4 Henderson Pl | Onehunga | Auckland 1061 | New Zealand
Switchboard: +49 2630 96520 | https://www.tomra.com

The information contained in this communication and any attachment is confidential and may be legally privileged. It should only be read by the person(s) to whom it is addressed. If you have received this communication in error, please notify the sender and delete the communication.
-----Original Message-----
From: Romain Virot
Sent: Thursday, 9 January 2020 01:55
To: etherlab-users at etherlab.org
Subject: [etherlab-users] DC drift issue

Hi,

After lurking on the list and trying many different things, I'm looking for some help.

I have issues with DC syncing in my setup. I'm using a Beckhoff CX5140, 4.4.112-rt127-lp151-rt kernel with all the patches from Gavin and 64-bit DC modules (so far one EL7041 and one ELM3004-0000 both from Beckhoff) at 1kHz (I've also tested 200Hz and 400 Hz with similar behavior).

I can successfully enter my cyclic loop, with the system being synchronized, but after some time (typ. few hundred seconds) the dmesg indicates many unmatched datagrams :

....
[14364.891659] EtherCAT WARNING: Datagram ffff88003f81d6d8 (domain0-0-main) was SKIPPED 1004 times.
[14365.712234] EtherCAT WARNING 0: 2000 datagrams UNMATCHED!
[14365.897276] EtherCAT WARNING: Datagram ffff88003f81d6d8 (domain0-0-main) was SKIPPED 1004 times.
[14366.713842] EtherCAT WARNING 0: 2000 datagrams UNMATCHED!
[14366.902893] EtherCAT WARNING: Datagram ffff88003f81d6d8 (domain0-0-main) was SKIPPED 1004 times.
[14367.715453] EtherCAT WARNING 0: 2000 datagrams UNMATCHED!
[14367.908512] EtherCAT WARNING: Datagram ffff88003f81d6d8 (domain0-0-main) was SKIPPED 1004 times.
[14368.717066] EtherCAT WARNING 0: 2000 datagrams UNMATCHED!
[14368.914130] EtherCAT WARNING: Datagram ffff88003f81d6d8 (domain0-0-main) was SKIPPED 1003 times.
[14369.718690] EtherCAT WARNING 0: 2 datagrams TIMED OUT!
[14369.718700] EtherCAT WARNING 0: 2000 datagrams UNMATCHED!
[14369.919746] EtherCAT WARNING: Datagram ffff88003f81d6d8 (domain0-0-main) was SKIPPED 1004 times.
[14370.720285] EtherCAT WARNING 0: 2000 datagrams UNMATCHED!
[14370.925364] EtherCAT WARNING: Datagram ffff88003f81d6d8 (domain0-0-main) was SKIPPED 1004 times.
[14371.721901] EtherCAT WARNING 0: 2000 datagrams UNMATCHED!
[14371.930982] EtherCAT WARNING: Datagram ffff88003f81d6d8 (domain0-0-main) was SKIPPED 1004 times.
[14372.723535] EtherCAT WARNING 0: 2000 datagrams UNMATCHED!
[14372.936607] EtherCAT WARNING: Datagram ffff88003f81d6d8 (domain0-0-main) was SKIPPED 1003 times.
[14373.725111] EtherCAT WARNING 0: 2 datagrams TIMED OUT!
....

My code is roughly set-up like this:

----- Before RT -----
ecrt_slave_config_dc(module->getSlaveConfig(ELM3004), 0x0700, 1000000, 500000, 20000, 0);   // ELM3004
ecrt_slave_config_dc(module->getSlaveConfig(EL7041), 0x0300, 1000000, 500000,, 0, 0); // EL7041

ecrt_master_select_reference_clock(master, network->getModule(EL7041); // Defines the EL7041 as the ref clock

----- Before the cyclic task -----

ecrt_master_set_send_interval(ethercatHandler->getMaster(), TIMESPEC2NS(cycletime) / 1000); // set the send interval to 1000 us clock_gettime(CLOCK_REALTIME, &wakeupTime); ecrt_master_application_time(ethercatHandler->getMaster(), TIMESPEC2NS(wakeupTime)); ecrt_master_sync_slave_clocks(ethercatHandler->getMaster());

----- Cyclic task -----

wakeupTime = timespec_add(wakeupTime, cycletime); clock_nanosleep(CLOCK_REALTIME, TIMER_ABSTIME, &wakeupTime, nullptr); // sleep until next cycle

ecrt_master_receive(ethercatHandler->getMaster());
ecrt_domain_process(ethercatHandler->getDomain());

.... (Emit signals for my commands) ...

ecrt_domain_queue(ethercatHandler->getDomain());

ecrt_master_reference_clock_time(ethercatHandler->getMaster(), timeForSync); ecrt_master_sync_slave_clocks(ethercatHandler->getMaster());

clock_gettime(CLOCK_REALTIME, &time);
*timeForSync = uint32_t(TIMESPEC2NS(time)); ecrt_master_application_time(ethercatHandler->getMaster(), *timeForSync);

ecrt_master_send(ethercatHandler->getMaster());

----- End of cyclic task -----

Note that the calculations inside the cyclic task are very small : I'm using slots//signal (QT5 framework) to treat commands (i.e. PDOs and 'RT' SDOs) which are dealt with on their own threads. I'm doing so because I don't care so much about the exact timing of my commands.

Any chance you can spot where my issue is coming from?

Thanks,

Romain

_______________________________________________
etherlab-users mailing list
etherlab-users at etherlab.org
https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.etherlab.org%2Fmailman%2Flistinfo%2Fetherlab-users&data=02%7C01%7Cgavin.lambert%40tomra.com%7Ccfb26e2eb7ae4b521d1d08d79439f34b%7C4308d118edd143008a37cfeba8ad5898%7C0%7C1%7C637140848926339128&sdata=jMYjFTeYXKFQ5114yi%2BC7xMVsurfAmKTdTOw9%2FZ9PLY%3D&reserved=0