[etherlab-users] Patch for Distributed Clock

[Graeme Foot]

Hi,

As part of my dabbling into getting my distributed clock stable I have
made a couple of changes.

I have attached two patches:
- etherlabmaster-1.5-2266-a_rtdm_dc.patch -> all my rtdm and dc changes
- etherlabmaster-1.5-2266-c_dc_clock_fix.patch -> just the dc clock
changes


My primary problem was an unstable master clock source.  In tracking
down what the problem was I also checked out what TwinCat was doing.

TwinCat has three distributed clock modes:
- (default) Master time/cycle updated to match ref slave time
- Ref slave time updated to match Master time
- Master time and Ref slave time updated to match an external clock
source

TwinCat also allows you to choose which slave to use as the ref slave.

TwinCat uses datagrams:
  ARMW  0x0  0x910      (sync slaves to ref)
  APWR  0x0  0x910      (sync ref to master)

and replaces the second datagram with a NOP when its not required.

The EtherLabs master uses datagrams:
  APWR  0x0  0x910      (sync ref to master)
  FRMW  0x1  0x910      (sync slaves to ref)

with no sync ref to master when its not required.  (ARMW vs FRMW
shouldn't make any difference.)


In my system I have two issues:
- My master clock (although it is now stable) gets a jitter of +-5000ns
(though usually +-2000ns) between the call to get the system time and
when the frame is actually sent out to the slaves.  This will be, I
assume, due to various code paths being called during the master_send.
- My Beckhoff CX1100-0004 coupler does not seem to be a stable ref
slave.

So what I've done is decided to use the default TwinCat method where I
pick a ref slave and update my master time based on the ref slave time.


What the patch includes:

1) ecrt_master_setup_distributed_clock()

Lets you set the masters application time and choose a ref slave (or
NULL) in one call.


2) ecrt_master_sync_slave_clocks_diff()

Sets the masters application time and returns the time difference
between the last master app time and the last ref slaves time (in one
call).

This gets called instead of ecrt_master_application_time();
ecrt_master_sync_reference_clock(); and ecrt_master_sync_slave_clocks(),
although these calls still work in their current fashion.


3) I have changed the ref_sync_datagram to 4 bytes instead of 8.

>From my reading of the ET1100 Hardware Data Sheet and looking at the
TwinCat datagrams the ref_sync_datagram should only sync the low 4 bytes
of the time.


4) I have changed ec_master_find_dc_ref_clock() to use either the user
specified ref-slave (if found and compatible) or the first compatible
slave (if user ref slave not specified or not found or compatible).


5) I have changed ec_master_find_dc_ref_clock() to also update the
ref_sync_datagram with the ref slave address.

Previously, if the first slave was not DC compatible (although unlikely)
then the ref_sync_datagram and sync_datagram would not refer to the same
slave and DC would fail to stay synced to the master.



How I'm using it (Note: I'm using my rtdm versions):

1) During setup I'm calling ecrt_master_setup_distributed_clock() and
supplying my desired reference slave (via its slave_config).


2) During realtime operation, directly before calling
ecrt_rtdm_master_send() (to reduce timing jitter) I'm getting the system
time then calling ecrt_rtdm_master_sync_slave_clocks_diff() and caching
the returned time difference.


3) After ecrt_rtdm_master_send() I'm using the cached time difference to
calculate a cycle period adjustment.  I then use the cycle period
adjustment to set a system time offset every period.  Whenever I read
the system time throughout the application I apply the system time
offset.

Instead of using rtai's rt_wait_period I instead use rt_sleep_until
using the offset system time for the next wakeup time.

I calculate my cycle period adjustment by averaging the time diff over
1000 periods.  I've found any filter shorter than that gets too much
variation.  I also add or subtract 1ns per period based on the current
cycles time diff to allow for any immediate variation requirements and
general drift.


4) If I have multiple masters (which I currently don't) then the first
master will act this way and all subsequent masters will use the current
method (by sending the master time to the ref slave).



One thing to note is that the time difference returned by
ecrt_rtdm_master_sync_slave_clocks_diff() is often one period out.  (ie
the time diff returned is ~1000000ns on a 1000000ns period.)  The code
is:

  // adjust slave time back to send time (ie remove transmission delay)
  uint32_t slaveTime  = EC_READ_U32(master->sync_datagram.data) - 
                        master->dc_ref_clock->transmission_delay;
  uint32_t masterTime = (uint32_t)master->app_time;
  
  *time_diff = masterTime - slaveTime;

  // set new app time
  ecrt_master_application_time(master, app_time);

This gets the time difference between the ref slaves time (adjusted for
transmission delay) from the returned sync datagram and the previous
master app time.  It then sets the new master app time.

In my application I don't care if I'm one or more cycles out, I just
care that the cycles stay in sync.  So I adjust my returned time diff to
the nearest cycle by:

  timeDiff = ((timeDiff + (scanTimeNS/2)) % scanTimeNS) -
(scanTimeNS/2);


Florian, a question for you, does this mean that the slaves might be
being set with a System Time Offset one period out?  This could account
for why the "Slave did not sync after 5000 ms" error occurs now and
then.  



To calculate my calibrated cpu frequency to match the ref slaves clock I
record the system time from when I start to get time diffs.  After some
time has passed I then perform the following calculation:

 int64_t diffTime = currentTime - startTime;
 int64_t adjTime  = diffTime + timeOffset;
 int64_t cpufreq  = nano2count(1000000000);

 cpu_freq = (int64_t)((double)cpufreq * (double)adjTime /
(double)diffTime);



Regards,
Graeme.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: etherlabmaster-1.5-2266-a_rtdm_dc.patch
Type: application/octet-stream
Size: 189724 bytes
Desc: etherlabmaster-1.5-2266-a_rtdm_dc.patch
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20120315/0e5999a4/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: etherlabmaster-1.5-2266-c_dc_clock_fix.patch
Type: application/octet-stream
Size: 14698 bytes
Desc: etherlabmaster-1.5-2266-c_dc_clock_fix.patch
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20120315/0e5999a4/attachment-0005.obj>