[etherlab-users] Distributed Clocks

Matthieu Bec mbec at gmto.org
Wed Mar 9 05:18:26 CET 2016


Another area maybe worth exploring if you haven't yet:
look for 'throttle' in the mailing list archive



Matthieu

On 3/8/16, 8:09 PM, "etherlab-users on behalf of Matthieu Bec" <etherlab-users-bounces at etherlab.org on behalf of mbec at gmto.org> wrote:


Hello Thomas,

These 13/50 are max latency numbers? both systems would qualify as reasonably good wrt. rt-preempt.

What I find surprising somehow: even your 'good' system shows so much variance: min: 6789, max: 50345
Shouldn't that be much always the same? Maybe something else is happening and amplified on the 'bad' system.

Regards,
Matthieu








On 3/8/16, 7:47 PM, "Thomas Bitsky Jr" <tbj at automateddesign.com> wrote:

Matthieu,

[snip]
in other words, you are positive the difference starts with ethercat master?

[/snip]

Yes. I have measured the time around that code block. That is where the variance occurs that causes me to have unmatched packets. 

That being said, one machine (i3) has a 13 microsecond latency, the other (celeron) has a 50 microsecond latency. If the ethercat userspace API is “sleeping” my process, then that would probably explain the problem. 

Are these problems avoided or reduced in the kernel space, or will I have the exact same timing?

Thanks!








On 3/8/16, 9:32 PM, "Matthieu Bec" <mbec at gmto.org> wrote:

>Hello Thomas,
>
>Both systems give similar cyclictest results? in other words, you are positive the difference starts with ethercat master?
>I'm not clear for example if celeron and i3 fall under the same processor family when you compile your kernel. Still such difference, explanation would seem far fetched.
>
>Regards,
>Matthieu
>
>
>
>
>On 3/8/16, 8:09 AM, "etherlab-users on behalf of Thomas Bitsky Jr" <etherlab-users-bounces at etherlab.org on behalf of tbj at automateddesign.com> wrote:
>
>Richard,
>
>I put a lot of time into your suggestions the past few days, and here is where I’m at.
>
>[snip]
>we use CLOCK_MONOTONIC for ecrt_master_application_time(), in fact we 
>use it thoughout. Probably not the solution, but worth a try.
>[/snip]
>
>
>As I generally do; thanks for catching that error for me.
>
>[snip]
>Once your cyclic test is stable (~10us max jitter), you may start 
>looking for delays in your code.
>[/snip]
>
>
>
>Semaphore and mutex calls were waking up inconsistently (the rt tests reported the same thing). I re-designed the API to eliminate the need for them in my code. I also reduced the memory footprint. Send, recv and timespec functions are the only system calls remaining.
>
>This brought the rest of the program down to very little execution time. I went through function by function, but couldn’t find the reason for the long execution times in my cleaned up code.
>
>However, I still have a surprisingly long execution time. The functions that cause the problem are from the ethercat master library.
>
>
>ecrt_master_receive(master_);
>	
>	
>
>	ecrt_domain_process(lrwDomainMgr_.domain);	
>	ecrt_domain_process(noLrwWriteDomainMgr_.domain);
>	ecrt_domain_process(noLrwReadDomainMgr_.domain);	
>
>
>
>I have tried this on two pieces of hardware. They run the same operating system and configuration, same EtherLAB master version, same Ethernet driver.
>
>System 1 (Intel i3 NUC, 1 Intel e1000e Ethernet port): exec min 6789 .. max 50345  (nano seconds)
>System 2 (Intel quad celeoron, 4 x e1000e Ethernet port): exec min 15588 … max 323376 (nano seconds)
>(Both systems: Ubuntu Server 14.04LTS, Linux core 3.12.50-rt68 #1 SMP PREEMPT RT)
>
>On system 1, the unmatched packets never happen; for whatever reason, those functions are returning much more rapidly. On the second system, the worst case happens about once per minute, once every two minutes. 
>
>1. Am I unreasonable in thinking these times are excessive? I understand that the system is communicating out to external hardware and there will be a delay, but the difference between the two computers is surprising to me.
>2. Is this a consequence of using the EtherCAT master from user space? I’ve always used it from kernel space before. However, I’ve never done as extensive of timing measurements before. The EtherLAB documentation mentioned that the userspace library only added 1 microsecond of overhead, if I recall. Is this as good as it gets from the user space?
>3. Is there a way to address this problem, or is it completely dependent upon the hardware?
>
>
>Thanks for any insights to the problem, and for all the advice everyone has submitted this far.
>
>
>
>
>
>
>
>On 3/3/16, 1:59 PM, "Richard Hacker" <ha at igh.de> wrote:
>
>>Hi,
>>
>>we use CLOCK_MONOTONIC for ecrt_master_application_time(), in fact we 
>>use it thoughout. Probably not the solution, but worth a try.
>>
>>- Richard
>>
>>On 03.03.2016 17:08, Thomas Bitsky Jr wrote:
>>>
>>> [snip]
>>> Are you using the dc_clock_adjust to modify your sleep time?  ie.
>>> something like
>>> [/snip]
>>>
>>>
>>> I’ve been trying, but the results are all over the place and I’m obviously not understanding what needs to be done with the clock.
>>>
>>> My latest attempt is has the scan time ticking from anywhere between 3930 and 4100 scans per second (reported by ethercat master command) and does nothing to eliminate skipped datagrams. I don’t think I’m fundamentally understanding what needs to be done.
>>>
>>> My latest code is below. If anyone has any insights, it would be greatly appreciated.
>>>
>>> ========
>>>
>>> int64_t  dc_adjust_ns = 0 ;
>>> int32_t  dc_diff_ns = 0;
>>>
>>>
>>>
>>> void calculateClockDrift(void) {
>>>      uint32_t ref_time = 0;
>>>      uint64_t prev_app_time = dc_time_ns;
>>>
>>>      dc_time_ns = system_time_ns();
>>>      // get reference clock time to synchronize master cycle
>>>      ecrt_master_reference_clock_time(master_, & ref_time);
>>>      dc_diff_ns = (uint32_t) prev_app_time - ref_time;
>>>      // calc drift (via un-normalised time diff)
>>>      int32_t delta = dc_diff_ns - prev_dc_diff_ns;
>>>      prev_dc_diff_ns = dc_diff_ns;
>>>
>>>
>>>      dc_diff_ns =
>>>          ((dc_diff_ns + (cycle_ns_ / 2)) % cycle_ns_) - (cycle_ns_ / 2);
>>>
>>> }
>>>
>>>
>>> #define TIMESPEC_ADD_NS(TS, NS)\
>>>      (TS).tv_nsec += (NS);\
>>> while ((TS).tv_nsec >= NANOS_PER_SEC) {\
>>>      (TS).tv_nsec -= NANOS_PER_SEC;\
>>>      (TS).tv_sec++;
>>> }
>>>
>>> #define TIMESPEC2NSEPOCH2000(T)\
>>>      ((uint64_t)(((T).tv_sec - 946684800 ULL) * 1000000000 ULL) + (T).tv_nsec)
>>>
>>> # define TON struct timespec# define TON_ENDTIME(MS)\
>>> time_add_ns((MS) * NANOS_PER_MILLISEC)
>>>
>>>
>>> static TON clockSyncTon_;
>>>
>>>
>>> int
>>> TON_ISDONE(struct timespec ts) {
>>>      struct timespec now;
>>>      clock_gettime(CLOCK_MONOTONIC, & now);
>>>      if (now.tv_sec > ts.tv_sec)
>>>          return 1;
>>>      else if (now.tv_sec == ts.tv_sec && now.tv_nsec >= ts.tv_nsec)
>>>          return 1;
>>>      else
>>>          return 0;
>>> }
>>>
>>>
>>> static bool
>>> wait_period(RtaiMain * inst) {
>>>
>>>      int rc;
>>>      bool done = false;
>>>      while (!done && inst - > doScan && runAll_) {
>>>          rc = clock_nanosleep(CLOCK_MONOTONIC,
>>>              TIMER_ABSTIME, & inst - > wakeupTime,
>>>              NULL);
>>>
>>>
>>>          if (rc == EFAULT) {
>>>              return false;
>>>          } else if (rc == EINTR) {
>>>              continue;
>>>          } else if (rc == EINVAL) {
>>>              return false;
>>>          } else {
>>>              done = 1;
>>>          }
>>>      }
>>>      TIMESPEC_ADD_NS(inst->wakeupTime,
>>> 	inst->cycleNs + dc_diff_ns );
>>>      return true;
>>>
>>> }
>>>
>>>
>>> static void
>>> cyclic_task(RtaiMain * inst) {
>>>
>>>      clock_gettime(CLOCK_MONOTONIC, & (inst - > wakeupTime));
>>>      /* start after one second */
>>>      inst - > wakeupTime.tv_sec++;
>>>      wait_period(inst);
>>>      while (runAll_ && inst - > doScan) {
>>>          //
>>>          // Trigger Fieldbus RX here.
>>>          //
>>>          //
>>>          ecrt_master_receive(master_);
>>>
>>>          // record the time we received the data so other parts of the program
>>>          // have an accurate time reading
>>>          globalTickTimeNs = ton_get_ns();
>>>
>>>          ecrt_domain_process(lrwDomainMgr_.domain);
>>>          ecrt_domain_process(noLrwWriteDomainMgr_.domain);
>>>          ecrt_domain_process(noLrwReadDomainMgr_.domain);
>>>
>>>          if (counter_) {
>>>
>>>              counter_—;
>>>          } else {
>>>              counter_ = 4000;
>>>
>>>              // check for master state
>>>              check_master_state();
>>>          }
>>>
>>>
>>>>>>          tick sub systems
>>>
>>>
>>>          //
>>>          // Trigger Fieldbus TX. This should be the last step
>>>          //
>>>          //
>>>          ecrt_domain_queue(lrwDomainMgr_.domain);
>>>          ecrt_domain_queue(noLrwWriteDomainMgr_.domain);
>>>          ecrt_domain_queue(noLrwReadDomainMgr_.domain);
>>>          clock_gettime(CLOCK_REALTIME, & dcTime_);
>>>          ecrt_master_application_time(
>>>              master_,
>>>              TIMESPEC2NSEPOCH2000(dcTime_));
>>>
>>>
>>>          if (TON_ISDONE(clockSyncTon_)) {
>>>              ecrt_master_sync_reference_clock(master_);
>>>              clockSyncTon_ = TON_ENDTIME(10); // milliseconds
>>>          }
>>>          ecrt_master_sync_slave_clocks(master_);
>>>
>>>          // send EtherCAT data
>>>          ecrt_master_send(master_);
>>>
>>> 	calculateClockDrift();
>>>
>>>
>>>          if (!wait_period(inst)) {
>>>              PRINT("%s Error with waiting! Stopping cyclic_task.\n",
>>>                  __FUNCTION__);
>>>              inst - > doScan = false;
>>>          }
>>>      }
>>>
>>> }
>>>
>

_______________________________________________
etherlab-users mailing list
etherlab-users at etherlab.org
http://lists.etherlab.org/mailman/listinfo/etherlab-users



More information about the etherlab-users mailing list