[etherlab-users] Control loop at higher frequencies
Jordan Palacios
jordan.palacios at pal-robotics.com
Wed Nov 13 16:45:50 CET 2019
Hi again.
Thank you both for your answers.
We've been waiting for the CU1128 and ET2000 to arrive before continuing
our tests. Here are our findings.
As for the CU1128 it is as both of you predicted, the improvement over the
CX-JC06 is negligible.
Using the ET2000 and Wireshark we could finally obtain a reliable measure
of the EtherCAT frame round trip time. The Time column is set to show time
since previous captured packet.
[image: et2000_capture.png]
This confirms the delay introduced by the slaves to be around 40us,
Then we tried to measure the delay caused by the network card. This has
proved to be tricky since we can't measure it reliably. In the end what we
did is finding how fast could we send consecutive frames. We did this by
gradually increasing the frequency of the control loop. Finally, for a
datagram of length 1216 (our full slave configuration) we can't go over
100us.
100us is the delay we assume for the network card. This delay is directly
related to the frame length, i.e., if we reduce the datagram length we can
go over 100us. We did some tests, and we have found out that if we can
manage to reduce to datagram to around 650 we can actually control at 4KHz
reliably.
By the way, here is some network card information.
EtherCAT: Master driver 1.5.2 551c87788b5e+
EtherCAT: 1 master waiting for devices.
e1000e 0000:03:00.0 eth0: removed PHC
e1000e: eth0 NIC Link is Down
e1000e 0000:00:19.0 eth1: removed PHC
ec_e1000e: EtherCAT-capable Intel(R) PRO/1000 Network Driver -
3.2.6-k-EtherCAT
ec_e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
ec_e1000e 0000:00:19.0: Interrupt Throttling Rate (ints/sec) set to
dynamic conservative mode
ec_e1000e 0000:00:19.0 0000:00:19.0 (uninitialized): registered PHC
clock
EtherCAT: Accepting 00:13:95:23:67:A1 as main device for master 0.
EtherCAT 0: Starting EtherCAT-IDLE thread.
ec_e1000e 0000:00:19.0 ecm0 (uninitialized): (PCI
Express:2.5GT/s:Width x1) 00:13:95:23:67:a1
ec_e1000e 0000:00:19.0 ecm0 (uninitialized): Intel(R) PRO/1000
Network Connection
ec_e1000e 0000:00:19.0 ecm0 (uninitialized): MAC: 10, PHY: 11, PBA
No: FFFFFF-0FF
ec_e1000e 0000:03:00.0: Interrupt Throttling Rate (ints/sec) set to
dynamic conservative mode
ec_e1000e 0000:03:00.0 0000:03:00.0 (uninitialized): registered PHC
clock
ec_e1000e 0000:03:00.0 eth0: (PCI Express:2.5GT/s:Width x1)
00:0c:8b:20:05:1f
ec_e1000e 0000:03:00.0 eth0: Intel(R) PRO/1000 Network Connection
ec_e1000e 0000:03:00.0 eth0: MAC: 3, PHY: 8, PBA No: FFFFFF-0FF
ec_e1000e: ecm0 NIC Link is Up 100 Mbps Full Duplex, Flow Control:
None
ec_e1000e 0000:00:19.0 ecm0 (uninitialized): 10/100 speed:
disabling TSO
So now we have switched to looking into reducing the datagram size. After
removing some non essential slaves we are left with 32 Elmo Gold Twitter
boards. With this configuration the datagram length is 1132 with LRW taking
1088.
This is the PDO mapping for each Elmo slave:
SM0: PhysAddr 0x1800, DefaultSize 140, ControlRegister 0x26,
Enable 1
SM1: PhysAddr 0x1900, DefaultSize 140, ControlRegister 0x22,
Enable 1
SM2: PhysAddr 0x1100, DefaultSize 32, ControlRegister 0x64,
Enable 1
RxPDO 0x1600 ""
PDO entry 0x607a:00, 32 bit, ""
PDO entry 0x60fe:01, 32 bit, ""
PDO entry 0x6040:00, 16 bit, ""
RxPDO 0x1607 ""
PDO entry 0x6060:00, 8 bit, ""
PDO entry 0x6071:00, 16 bit, ""
RxPDO 0x160e ""
PDO entry 0x6073:00, 16 bit, ""
SM3: PhysAddr 0x1180, DefaultSize 32, ControlRegister 0x20,
Enable 1
TxPDO 0x1a07 ""
PDO entry 0x6041:00, 16 bit, ""
PDO entry 0x6061:00, 8 bit, ""
PDO entry 0x6078:00, 16 bit, ""
PDO entry 0x6064:00, 32 bit, ""
PDO entry 0x606c:00, 32 bit, ""
PDO entry 0x2205:01, 16 bit, ""
PDO entry 0x20a0:00, 32 bit, ""
We use 15 bytes for RxPDO and 19 bytes for TxPDO. This adds to 34 bytes for
each slave. 34 * 32 = 1088 which is our current LRW.
AFAIK shouldn't the input and output share the same space in the datagram?
We expected the size used by each slave to be max(RxPDO, TxPDO) instead of
the sum.
At least this is what we understand from EtherCAT documentation.
[image: FMMU_Mapping.png]
We did a quick setup using TwinCAT and the LRW is in fact optimized there.
On our application we use ecrt_slave_config_pdos for PDO configuration. I
looked into etherlab documentation and the rest of the PDO configuration
functions but could not find nothing related to this.
If the space used for input and output could be shared our issues would be
solved. Maybe we are doing something wrong?
Once again, thanks for your help.
Kind regards.
Jordán.
On Fri, 1 Nov 2019 at 00:04, Gavin Lambert <gavin.lambert at tomra.com> wrote:
> FYI, the CU1128 has three internal ESCs:
>
> Due to the internal port ordering, the packets traverse the ports in their
> numeric order (a packet arriving at 1 will be sent to 2; when it returns
> back through 2 it will be sent to 3, etc).
>
>
>
> I haven’t personally tested the port latency of the CU1128, but I think I
> remember it being negligible.
>
>
>
> Regarding using a switch for monitoring: the only configuration which is
> officially supported is to put the monitoring switch between the master and
> first slave. (The entire slave network is conceptually one big Ethernet
> device.) Doing it elsewhere may or may not work depending on your
> particular layout, and it will certainly cause higher latency (and some
> switches are better than others).
>
>
>
>
> *Gavin Lambert*Senior Software Developer
>
>
> [image: TOMRA] <http://www.compacsort.com> [image: Facebook]
> <https://www.facebook.com/Compacsort> [image: Linkedin]
> <https://www.linkedin.com/company/compac-sorting-equipment/> [image:
> Youtube] <https://vimeo.com/compacsort> [image: twitter]
> <https://twitter.com/compacsort> [image: instagram]
> <https://www.instagram.com/compacsort/>
>
> *COMPAC SORTING EQUIPMENT LTD* | 4 Henderson Pl | Onehunga | Auckland
> 1061 | New Zealand
> Switchboard: +64 96 34 00 88 | tomra.com <http://www.tomra.com>
>
> The information contained in this communication and any attachment is
> confidential and may be legally privileged. It should only be read by the
> person(s) to whom it is addressed. If you have received this communication
> in error, please notify the sender and delete the communication.
>
> *From:* Graeme Foot
> *Sent:* Friday, 1 November 2019 11:20
> *To:* Jordan Palacios <jordan.palacios at pal-robotics.com>
> *Cc:* etherlab-users at etherlab.org
> *Subject:* Re: [etherlab-users] Control loop at higher frequencies
>
>
>
> Hi Jordan,
>
>
>
> The GX-JC06 is a 6 port junction. The timings with and without the
> GX-JC06 look about correct. Just to be sure you know how they work, the
> EtherCAT frames are passed to each port of the junction sequentially, not
> in parallel. It will also pass each port of the junction regardless of
> whether it is open or not, with a time overhead per port. I don't have the
> documentation for the GX-JC06, but here is a diagram from the EK1100:
>
>
>
>
>
> So with your example where one branch is connect direct to the master vs
> via the GX-JC06 I would expect the time to increase by approx. 620ns per
> MII port and 135ns per EBUS port (from your info below). It looks like the
> junction is made up of two 4 port chips joined by the EBUS ports, something
> like this:
>
>
>
>
>
> So you need to add the overhead of:
>
> - branch slave return to CX-JC06 (1 x 620ns)
>
> - EBUS to EBUS link (2 x 135)
>
> - MII ports (8 x 620)
>
>
>
> That adds up to approx. 4610ns extra, totalling 8070ns (within ballpark of
> your 8220ns value, also based on a few assumptions).
>
>
>
> Adding each new branch will add the overhead of the entire branch (the
> time taken for the frame to get to the last slave of the branch, and
> return).
>
>
>
> I would expect the a Beckhoff CU1128 to behave very similarly, except that
> it sounds like the per port overhead is slightly slower (1000ns vs 620ns).
>
>
>
>
>
> I'm still picking it's something at the master / device driver end.
>
>
>
> Maybe you could check the ethercat version of the e1000e driver is in use
> correctly. I'm still running kernel 2.6.32 and connect via a CX2100 so not
> sure if there's any differences to your kernel and using the e1000e, but
> from my dmesg I can see:
>
> EtherCAT: 1 master waiting for devices.
>
> ec_cx2100: EtherCAT-capable Beckhoff CX2100 EtherCAT Network Driver
> (Revision 2)
>
> ec_cx2100: Found EtherCAT Master (0x0014), Revision: 0x0000, txDMA: 0x01,
> rxDMA: 0x00
>
> ec_cx2100: EtherCAT Master, MAC address: 00:01:05:13:7e:d0, Link: Connected
>
> EtherCAT: Accepting 00:01:05:13:7E:D0 as main device for master 0.
>
>
>
> I do also have the ec_e1000e driver for the other ports...
>
> ec_e1000e: Ethercat-capable Intel(R) PRO/1000 Network Driver -
> 1.0.2-k2(ethercat)
>
>
>
> Doing a "lspci -k" also shows that the ethercat versions of the drivers
> are correctly loaded:
>
> 02:00.0 Class 0200: 8086:10d3 ec_e1000e
>
> 03:00.0 Class 0200: 8086:10d3 ec_e1000e
>
> 04:00.0 Class ff00: 15ec:5000 ec_cx2100
>
>
>
> Not sure what else to check for.
>
>
>
> Regards,
>
> Graeme.
>
>
>
>
>
> *From:* Jordan Palacios <jordan.palacios at pal-robotics.com>
> *Sent:* Friday, 1 November 2019 1:23 AM
> *To:* Gavin Lambert <gavin.lambert at tomra.com>
> *Cc:* Graeme Foot <Graeme.Foot at touchcut.com>; etherlab-users at etherlab.org
> *Subject:* Re: [etherlab-users] Control loop at higher frequencies
>
>
>
> Hi,
>
> Thanks both of you for your answers. I'll try to address all your points.
>
> We run our master in a PC with Ubuntu 16.04. Kernel 4.9.115 with the
> RT_PREEMPT patch. The network driver used by the master is the e1000e. Here
> are the details of the network card:
>
>
>
> description: Ethernet interface
> product: 82579LM Gigabit Network Connection
> vendor: Intel Corporation
> physical id: 19
> bus info: pci at 0000:00:19.0
> logical name: eth1
> version: 04
> serial: 00:13:95:2e:e1:b0
> capacity: 1Gbit/s
> width: 32 bits
> clock: 33MHz
> capabilities: bus_master cap_list ethernet physical tp 10bt 10bt-fd
> 100bt 100bt-fd 1000bt-fd autonegotiation
> configuration: autonegotiation=on broadcast=yes driver=e1000e
> driverversion=3.2.6-k firmware=0.15-4 latency=0 link=no multicast=yes
> port=twisted pair
> resources: irq:26 memory:f7f00000-f7f1ffff memory:f7f35000-f7f35fff
> ioport:f080(size=32)
>
> The slaves are connected in a star topology using an OMROM GX-JC06
> EtherCAT Junction
> <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.ia.omron.com%2Fproducts%2Ffamily%2F3079%2Ffeature.html&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925415672&sdata=iUjP2fjPUalE4P7feAXsfXmWj6gUej3zi5x0pviijUw%3D&reserved=0>.
> I didn't mention this in my previous mail but we had detected that this
> junction does indeed add considerable delay. For example, if we connect one
> of the branches directly to the control PC we measure a DC system time
> transmission delay of 3460ns. When the same branch is connected to the
> junction with no other slaves reports 8220ns. This gets worse when we start
> adding more slaves in different ports until we reach the 38995ns I wrote in
> my first mail.
>
>
>
> Here is the info from the two slaves that comprise the junction:
>
>
>
> === Master 0, Slave 0 ===
> Device: Main
> State: PREOP
> Flag: +
> Identity:
> Vendor Id: 0x00000083
> Product code: 0x00000064
> Revision number: 0x00010000
> Serial number: 0x00000000
> DL information:
> FMMU bit operation: no
> Distributed clocks: yes, 64 bit
> DC system time transmission delay: 0 ns
> Port Type Link Loop Signal NextSlave RxTime [ns] Diff [ns]
> NextDc [ns]
> 0* MII up open yes - 1316003530 0
> 0
> 1 MII up open yes 8 1316019530 16000
> 620
> 2 EBUS up open yes 15 1316043660 40130
> 135
> 3 MII up open yes 1 1316011570 8040
> 640
> General:
> Group: Junction Slave
> Image name:
> Order number: GX-JC06(IN,X2,X3)
> Device name: GX-JC06(IN,X2,X3)�@6�|�[�g�����X���[�u
> Flags:
> Enable SafeOp: no
> Enable notLRW: no
> Current consumption: 0 mA
>
>
>
> === Master 0, Slave 15 ===
> Device: Main
> State: PREOP
> Flag: +
> Identity:
> Vendor Id: 0x00000083
> Product code: 0x00000065
> Revision number: 0x00010000
> Serial number: 0x00000000
> DL information:
> FMMU bit operation: no
> Distributed clocks: yes, 64 bit
> DC system time transmission delay: 16135 ns
> Port Type Link Loop Signal NextSlave RxTime [ns] Diff [ns]
> NextDc [ns]
> 0* EBUS up open yes 0 1320871430 0
> 135
> 1 MII up open yes 24 1320892930 21500
> 620
> 2 MII up open yes 35 1320895290 23860
> 620
> 3 MII up open yes 16 1320880610 9180
> 630
> General:
> Group: Junction Slave
> Image name:
> Order number: GX-JC06(X4,X5,X6)
> Device name: GX-JC06(X4,X5,X6)�@6�|�[�g�����X���[�u
> Flags:
> Enable SafeOp: no
> Enable notLRW: no
> Current consumption: 0 mA
>
>
>
> We are now in the process of switching to a Beckhoff CU1128
> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.beckhoff.com%2FCU1128%2F&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925425665&sdata=0ZOOOYmK9NVE6wT1fnR2Srn6Iz%2FKysXmusyx5XPM6fg%3D&reserved=0>.
> Their support has assured us that the delay introduced should be around 1us
> per port used and should remain constant independently of the number of
> slaves and frame size.
>
>
>
> We tried a setup with a switch in the middle to capture the network
> traffic. Unfortunately we couldn't make it work. The master would simply
> fail to find any slaves in the network the moment the switch was plugged
> in. Even when there was no other PC/laptop connected to the switch to sniff
> the traffic. We'll give it another go.
>
>
>
> We also took additional measures using two Beckhoff FB1111-0142
> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.beckhoff.ee%2Fenglish.asp%3Fethercat%2Ffb1111.htm&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925425665&sdata=i1OB9wIFFwBMnqJEpYDTY2NBCdI4y4n9EjkvmW9ekec%3D&reserved=0>.
> We connected one of them before the switch and the other after the last
> slave. Then took the measure of the delay between the signal of the SOF
> pins. The result was a delay of around 39.2us, something really close to
> the value reported by the DC system time transmission delay. Using
> this approach we took measures of different sections of the bus. So now we
> have an idea of the average delay introduced by each slave. This further
> confirmed the delay introduced by the OMROM junction.
>
>
>
> The delay introduced by the PC network card is something we
> definitely haven't taken into account and may explain why we don't receive
> the frames on time. I don't think it as much as 550us though. If it was it
> wouldn't work at 2Khz neither.
>
>
>
> Kind regards.
>
>
>
> Jordán.
>
>
>
> On Thu, 31 Oct 2019 at 00:07, Gavin Lambert <gavin.lambert at tomra.com>
> wrote:
>
> When you use the master debug interface (or the newly added “ethercat
> pcap” command, although you won’t have that in your version yet), the
> receive timestamps are the time that they are received by the master, which
> is the time that you called ecrt_master_receive.
>
>
>
> You can get a better understanding of the “real” wire transfer time by
> connecting a separate monitoring device between the master and first slave
> – a proper network monitor is ideal, but you can made do with a standard
> Ethernet hub/switch with three ports connected (master, first slave,
> monitoring laptop running Wireshark), although for best results (especially
> if it’s a Windows laptop) you should disable all the network protocols on
> the laptop adapter other than the minimum it needs to do the Wireshark
> monitoring, so that it doesn’t inadvertently transmit packets of its own.
> (These won’t hurt the network itself, since both the master and slave will
> drop non-EtherCAT packets, but it may worsen the latency.) This will still
> be somewhat inaccurate since it’s still timestamping when the packet was
> processed rather than when it was actually received in hardware, but it’s
> likely to be more accurate than doing it on the master since it’s not
> waiting for the cycle delay.
>
>
>
> As Graeme said, you can also put the monitoring device elsewhere in the
> network (all packets will always pass every point in the network – except
> the end), although then it becomes harder to meaningfully use the
> timestamps.
>
>
>
> *Gavin Lambert*
> Senior Software Developer
>
> [image: cid:image001.png at 01D5909F.79B9FED0]
> [image: TOMRA]
> <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.compacsort.com%2F&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925435660&sdata=722bhEPG34D1dhARt85nCL6rNxiiKxV4LWnSgiJrWy8%3D&reserved=0>[image:
> Facebook]
> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FCompacsort&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925435660&sdata=aB6s9D3qko%2BUlSwjVXaxoNpvm5hEsOuz7nQGo%2FSQ6dQ%3D&reserved=0>[image:
> Linkedin]
> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fcompac-sorting-equipment%2F&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925445655&sdata=vaV4nLLGbN4FFVvjo4yoBWJiUhfTH118GubqO38S8ow%3D&reserved=0>[image:
> Youtube]
> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvimeo.com%2Fcompacsort&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925445655&sdata=OAqUWZKe1WnHtAPFgWwsRgLUc5gJW%2BpdGw1fmtsOOzA%3D&reserved=0>[image:
> twitter]
> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fcompacsort&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925455650&sdata=hW3lpDVxYRao8GnIjZADw8cp2bx07wMsssusLNV9BxY%3D&reserved=0>[image:
> instagram]
> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.instagram.com%2Fcompacsort%2F&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925455650&sdata=IIST0Agy%2B8En9sjbgKFmAqq4v%2FbT4glLvzgnbOXpi8s%3D&reserved=0>
>
> *COMPAC SORTING EQUIPMENT LTD* | 4 Henderson Pl | Onehunga | Auckland
> 1061 | New Zealand
> Switchboard: +64 96 34 00 88 | tomra.com
> <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.tomra.com&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925455650&sdata=jOCLm%2FJOfHMaC44rpWmUgKKPxaWa2cAguxJHHIVH0t4%3D&reserved=0>
>
> The information contained in this communication and any attachment is
> confidential and may be legally privileged. It should only be read by the
> person(s) to whom it is addressed. If you have received this communication
> in error, please notify the sender and delete the communication.
>
> *From:* Graeme Foot
> *Sent:* Thursday, 31 October 2019 11:49
> *To:* Jordan Palacios <jordan.palacios at pal-robotics.com>
> *Cc:* etherlab-users at etherlab.org
> *Subject:* Re: [etherlab-users] Control loop at higher frequencies
>
>
>
> Hi,
>
>
>
> It is sounding like the data time on the wire is taking too long (> 250us).
>
>
>
> Besides doubling the "DC system time transmission delay" of the last
> slave, you can also check the "Diff [ns]" value of your first slave. This
> may give you a more accurate idea if you have a star topology, as the
> return trip of the frame from the last slave bypasses the fingers of the
> stars. e.g.:
>
>
>
> ethercat slaves -v -p0
>
>
>
> === Master 0, Slave 0 ===
>
> Alias: 10001
>
> Device: Main
>
> State: OP
>
> Flag: +
>
> Identity:
>
> Vendor Id: 0x00000002
>
> Product code: 0x04562c52
>
> Revision number: 0x00110000
>
> Serial number: 0x00000000
>
> DL information:
>
> FMMU bit operation: no
>
> Distributed clocks: yes, 64 bit
>
> DC system time transmission delay: 0 ns
>
> Port Type Link Loop Signal NextSlave RxTime [ns] Diff [ns]
> NextDc [ns]
>
> 0* EBUS up open yes - 3638440690
> 0 0
>
> 1 MII up open yes 1 3638442410 *1720*
> 560
>
> 2 N/A down closed no - -
> - -
>
> 3 N/C down closed no - -
> - -
>
>
>
> You will also need to add on 2 * "master to first slave transmission
> delay" which you will probably need to guess. If you use a Beckhoff CX2020
> (or similar) where the CX2100 is directly connected to the EBus this will
> likely be around 150us. If your master has an ethernet port and the first
> slave is an amp or a coupler this may be around 550us or more (depending on
> cable length). You could also calc the frame transmission time. Your
> frame length is 782 bytes. So that should take ~ 6 - 7us. Plus whatever
> other overheads the ethernet card / driver has.
>
>
>
> Reducing the number of configured slaves (even with them still physically
> plugged in) will reduce the ec_master_domain_queue() and ec_master_send()
> overhead time, getting the frame to the wire quicker. It will also reduce
> the frame length and the related frame transmission time (especially if you
> remove slaves with bigger datagram overhead). It looks like this is enough
> to result in the total frame roundtrip time being reduce enough for it to
> return and be ready by the time that ec_master_receive() is called.
>
>
>
>
>
> Just FYI, I have a machine downstairs at the moment with 42 slaves (14 of
> which are amps), linear topology. The last slaves transmission delay is
> 19560ns. The first slaves Diff is 39390ns. The EtherCAT frame size is 886
> bytes. The ec_master_receive() to ec_master_send() time is approx 23us.
> So your values sound in line with what I have, except that the wireshark
> output is showing a large cycle time.
>
>
>
> In general it's looking like the time on the wire is taking longer than it
> should. How are you capturing the EtherCAT frame data? e.g.: a switch
> between your master and first slave? If so, try moving it further down the
> chain, doing an "ethercat rescan" and check the transmission delays. See
> if the switch is causing a large delay between those slaves.
>
>
>
> What is your network card and driver used by the master? What is your
> realtime system? If you are using tshark on the master PC there might be
> delays being introduced in the network card driver.
>
>
>
>
>
> Regards,
>
> Graeme.
>
>
>
>
>
> *From:* etherlab-users <etherlab-users-bounces at etherlab.org> *On Behalf
> Of *Jordan Palacios
> *Sent:* Thursday, 31 October 2019 4:36 AM
> *To:* etherlab-users at etherlab.org
> *Subject:* [etherlab-users] Control loop at higher frequencies
>
>
>
> Hi.
>
>
>
> We've been working with the etherlab master for some time now. On our
> current setup we have around 40 slaves and our control loop runs at 1Khz
> without any issues. We use the stable version of etherlab with the 20180622
> patchset.
>
>
>
> Recently we've tried increasing the loop frequency to achieve a better
> control. The target frequency is 4Khz. After some troubles we managed a
> stable control at 2Khz. Then we went for the 4Khz and this is where we have
> hit a roadblock.
>
>
>
> A warning like this is generated each second:
>
>
>
> [11547.183302] EtherCAT WARNING 0: 4000 datagrams UNMATCHED!
> [11547.619399] EtherCAT WARNING: Datagram ffff88040b6bf318
> (domain0-0-main) was SKIPPED 4004 times.
>
>
>
> As per what Florian explained here
> <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.etherlab.org%2Fpipermail%2Fetherlab-users%2F2009%2F000386.html&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925465640&sdata=nnp5ttQXw6WxFh1JRHA7McEawbXbjTocQizHvGSsOqo%3D&reserved=0> this
> means that the answer to the last datagram sent has not been received yet.
> I don't think this is related to the rate at which we execute the control
> loop. We have instrumented it and is steady at 250us with a jitter below
> 10us (see attachment). Furthermore, we have also enabled the ethercat
> master debug interface. Here is a sample of the traffic output using tshark:
>
>
>
> 7807 0.975750173 MS-NLB-PhysServer-19_95:2e:e1:b0 → Broadcast ECAT 810
> 'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 69
> 7808 0.975757725 Congatec_2e:e1:b0 → Broadcast ECAT 810 'LRW': Len:
> 782, Adp 0x0, Ado 0x0, Wc 0
> 7809 0.976000284 MS-NLB-PhysServer-19_95:2e:e1:b0 → Broadcast ECAT 810
> 'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 69
> 7810 0.976007872 Congatec_2e:e1:b0 → Broadcast ECAT 810 'LRW': Len:
> 782, Adp 0x0, Ado 0x0, Wc 0
> 7811 0.976252515 MS-NLB-PhysServer-19_95:2e:e1:b0 → Broadcast ECAT 810
> 'LRW': Len: 782, Adp 0x0, Ado 0x0, Wc 69
> 7812 0.976260050 Congatec_2e:e1:b0 → Broadcast ECAT 810 'LRW': Len:
> 782, Adp 0x0, Ado 0x0, Wc 0
>
>
>
> The master received is done right at the beginning of the loop, and is
> followed by the domain process, domain queue and master send. The
> calculations are done after. We instrumented the received/send calls and we
> know together take around 20us.
>
>
>
> We originally made our calculations before the domain queue and master
> send. After reading what Graeme explains here
> <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.etherlab.org%2Fpipermail%2Fetherlab-users%2F2018%2F003351.html&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925465640&sdata=fcw4Z8OP9LjEGl66LCHN4C6y6V5oHKWYjpAOPjaMyi4%3D&reserved=0> we
> changed the order to allow for more time for the data in the wire. This
> change helped achieving the 2Khz.
>
>
>
> All of this means that the time at which the frames are received is
> another good indicator of the correct control loop periodicity. Note also
> how the next frame is sent right after.
>
>
>
> Thing is, even though the WC is the expected (69 in this case), we think
> that these frames are always late. The datagram received is the one of the
> previous cycle. Hence the constant warning of 4000 datagrams skipped. The
> statistics reported by the "ethercat master" command show 4000 frames
> transmitted/received per second and no frames are lost.
>
>
>
> Checking the "DC system time transmission delay" of the last slave we get
> a value of 38995 ns, which would translate to something like 80us for the
> frame to be in the wire. Keeping in mind we are allowing for 230us for the
> datagram to return (250us cycle minus 20us for receive/send) it should be
> more than enough. Yet something doesn't add up.
>
>
>
> The only thing that seems to help is reducing the amount of slaves. If we
> switch to something like a 25 slave configuration then we manage to control
> at 4KHz without issues. We don't physically disconnect them though so
> the DC system time transmission delay remains the same. Obviously frame
> size is smaller.
>
>
>
> We are not performing any calls related to distributed clocks in the
> master. If I understand correctly DC are used for synchronizing the data
> processing of the slaves. It should not affect the frame delay, right?
>
>
>
> Any ideas are appreciated.
>
>
>
> Kind regards.
>
>
>
> Jordán.
>
>
>
> --
>
> *Jordán Palacios*
>
> Software Engineer
>
>
>
> [image:
> https://lh6.googleusercontent.com/mgsVJlQ9p5jN1eI-8SP0YOxUhv0BEXrjZFnqqzYgaopfqhinUDbC7abJqhSV9RmgauHBXoff-GMTrJpOL2B9iYbSUgQN3cY8gIFI1XzcMdqZ6nG_Qera_i9qg2VQk35dG0uR01Ut]
>
> C/ Pujades 77-79, 4-4
>
> 08005 Barcelona, Spain
>
>
>
> *Skype *jordanpalacios.pal-robotics
>
> *Tel *+34 93 414 53 47
>
> jordan.palacios at pal-robotics.com
>
>
>
> www.pal-robotics.com
> <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pal-robotics.com%2F&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925475640&sdata=AmoZZqoZOYFraH8etFt5YcNDj9whYEHDyRm3kM%2FGZ8E%3D&reserved=0>
>
> Facebook
> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2Fpalrobotics%2F&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925475640&sdata=k9N87N0zwHiwmw7T3luUxqrSi6Xj4%2BjMI08QGHSQsOE%3D&reserved=0>
> | Twitter
> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FPALRobotics&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925485629&sdata=Z6%2BF4jbaxy9fUPdVmdK9awVdCOz3p28RnXrt28qVqLA%3D&reserved=0>
> | YouTube
> <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.youtube.com%2Fuser%2FPALRobotics&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925485629&sdata=d76uC64hGSH5HzUWRiSLQ22VjA6dk4lMgevXaIFxi5Q%3D&reserved=0>
> | Blog
> <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fblog.pal-robotics.com%2F&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925495625&sdata=POUusUe1wg4rgFb%2BCJWVNNOfrd%2FToQNu2D605hkOqN4%3D&reserved=0>
>
>
>
> AVISO DE CONFIDENCIALIDAD: Este mensaje y sus documentos adjuntos, pueden
> contener información privilegiada y/o confidencial que está dirigida
> exclusivamente a su destinatario. Si usted recibe este mensaje y no es el
> destinatario indicado, o el empleado encargado de su entrega a dicha
> persona, por favor, notifíquelo inmediatamente y remita el mensaje original
> a la dirección de correo electrónico indicada. Cualquier copia, uso o
> distribución no autorizados de esta comunicación queda estrictamente
> prohibida.
>
>
>
> CONFIDENTIALITY NOTICE: This e-mail and the accompanying document(s) may
> contain confidential information which is privileged and intended only for
> the individual or entity to whom they are addressed. If you are not the
> intended recipient, you are hereby notified that any disclosure, copying,
> distribution or use of this e-mail and/or accompanying document(s) is
> strictly prohibited. If you have received this e-mail in error, please
> immediately notify the sender at the above e-mail address.
>
>
>
>
> --
>
> *Jordán Palacios*
>
> Software Engineer
>
>
>
> [image:
> https://lh6.googleusercontent.com/mgsVJlQ9p5jN1eI-8SP0YOxUhv0BEXrjZFnqqzYgaopfqhinUDbC7abJqhSV9RmgauHBXoff-GMTrJpOL2B9iYbSUgQN3cY8gIFI1XzcMdqZ6nG_Qera_i9qg2VQk35dG0uR01Ut]
>
> C/ Pujades 77-79, 4-4
>
> 08005 Barcelona, Spain
>
>
>
> *Skype *jordanpalacios.pal-robotics
>
> *Tel *+34 93 414 53 47
>
> jordan.palacios at pal-robotics.com
>
>
>
> www.pal-robotics.com
> <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pal-robotics.com%2F&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925495625&sdata=4yR5VDjrxzFcgmqcwsueoa%2FWNThlWBxdTogEDvYZ%2Faw%3D&reserved=0>
>
> Facebook
> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2Fpalrobotics%2F&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925505620&sdata=bcgyXwL6jeFRGbg1s1IIWYfG4Pp0D7T7abI1q2M5lgU%3D&reserved=0>
> | Twitter
> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FPALRobotics&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925505620&sdata=sNd9TRlb2YEmpmlI8tKkdwxNSeSdxTs3opzgSAQ3uLY%3D&reserved=0>
> | YouTube
> <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.youtube.com%2Fuser%2FPALRobotics&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925515611&sdata=lg6SjqNM6kzMEKRSE0C1%2BoSjoGP1uqcZ2FrCf12Afo4%3D&reserved=0>
> | Blog
> <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fblog.pal-robotics.com%2F&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cd68e6d7539a148554b9908d75e507243%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637081571925515611&sdata=CsjrirWrmi%2FdFVAvMKMGx5FNae%2BJZ5ZbR%2FQ4jJqiBp4%3D&reserved=0>
>
>
>
> AVISO DE CONFIDENCIALIDAD: Este mensaje y sus documentos adjuntos, pueden
> contener información privilegiada y/o confidencial que está dirigida
> exclusivamente a su destinatario. Si usted recibe este mensaje y no es el
> destinatario indicado, o el empleado encargado de su entrega a dicha
> persona, por favor, notifíquelo inmediatamente y remita el mensaje original
> a la dirección de correo electrónico indicada. Cualquier copia, uso o
> distribución no autorizados de esta comunicación queda estrictamente
> prohibida.
>
>
>
> CONFIDENTIALITY NOTICE: This e-mail and the accompanying document(s) may
> contain confidential information which is privileged and intended only for
> the individual or entity to whom they are addressed. If you are not the
> intended recipient, you are hereby notified that any disclosure, copying,
> distribution or use of this e-mail and/or accompanying document(s) is
> strictly prohibited. If you have received this e-mail in error, please
> immediately notify the sender at the above e-mail address.
>
--
Jordán Palacios
Software Engineer
C/ Pujades 77-79, 4-4
08005 Barcelona, Spain
Skype jordanpalacios.pal-robotics
Tel +34 93 414 53 47
*jordan.palacios at pal-robotics.com <jordan.palacios at pal-robotics.com>*
www.pal-robotics.com
Facebook <https://www.facebook.com/palrobotics/> | Twitter
<https://twitter.com/PALRobotics> | YouTube
<http://www.youtube.com/user/PALRobotics> | Blog
<http://blog.pal-robotics.com/>
AVISO DE CONFIDENCIALIDAD: Este mensaje y sus documentos adjuntos, pueden
contener información privilegiada y/o confidencial que está dirigida
exclusivamente a su destinatario. Si usted recibe este mensaje y no es el
destinatario indicado, o el empleado encargado de su entrega a dicha
persona, por favor, notifíquelo inmediatamente y remita el mensaje original
a la dirección de correo electrónico indicada. Cualquier copia, uso o
distribución no autorizados de esta comunicación queda estrictamente
prohibida.
CONFIDENTIALITY NOTICE: This e-mail and the accompanying document(s) may
contain confidential information which is privileged and intended only for
the individual or entity to whom they are addressed. If you are not the
intended recipient, you are hereby notified that any disclosure, copying,
distribution or use of this e-mail and/or accompanying document(s) is
strictly prohibited. If you have received this e-mail in error, please
immediately notify the sender at the above e-mail address.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0003.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image012.png
Type: image/png
Size: 31482 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0076.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image013.png
Type: image/png
Size: 51254 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0077.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image014.png
Type: image/png
Size: 11438 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0078.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image015.png
Type: image/png
Size: 1629 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0079.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image016.png
Type: image/png
Size: 1750 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0080.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image017.png
Type: image/png
Size: 1855 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0081.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image018.png
Type: image/png
Size: 1970 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0082.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image019.png
Type: image/png
Size: 20278 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0083.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image020.png
Type: image/png
Size: 1506 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0084.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 18857 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0085.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logo_compac_5dcf97ef-52f5-498c-8b9b-728410ddffaf.png
Type: image/png
Size: 11438 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0086.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: compacicon_82e8a8c7-154a-4a32-9720-a5badb6258e0.png
Type: image/png
Size: 1629 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0087.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: facebook_fa85b924-53b9-45cc-8162-0564f64ec3a3.png
Type: image/png
Size: 1750 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0088.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: linkedin_4ec016ad-84fa-443c-85a3-b9615a4ccef8.png
Type: image/png
Size: 1855 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0089.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: youtube_32142163-fc27-4aed-b14d-e8a377f98a6d.png
Type: image/png
Size: 1970 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0090.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: twitter_d89338d8-98c8-4b65-9a9e-7b1333160b0d.png
Type: image/png
Size: 20278 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0091.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: insta2_1cd85de9-b3a2-4971-9904-52b2481a7c82.png
Type: image/png
Size: 1506 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0092.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: et2000_capture.png
Type: image/png
Size: 128040 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0093.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: FMMU_Mapping.png
Type: image/png
Size: 160623 bytes
Desc: not available
URL: <http://lists.etherlab.org/pipermail/etherlab-users/attachments/20191113/2aa0bf08/attachment-0094.png>
More information about the Etherlab-users
mailing list