[Etherlab-users] Running a large number of slaves
Geller, Nir
nir.geller at servotronix.com
Thu Apr 29 16:30:13 CEST 2021
Hi Gavin,
I'm sorry to be a pain about this.
When developing our ethercat slaves we are using TI's Sitara AM5728. Long ago we've confirmed that this chip works only with overlapping PDOs, and this can't be configured, so unfortunately I can't disable it.
If I do not invoke
ecrt_slave_config_overlapping_pdos(sc, 1)
A malformed packet returns from the ecat slave, and the application fails....
However, we've been working successfully with etherlab's ethercat master and the AM5728 for more than 2 years now. Overlapping PDOs enabled.
I managed to setup 7 ecat slaves, each with a PDO domain of 237 bytes.
All the PDO entries are separate and unique objects.
When running 6 slaves the PDO domain sums up to 1422 bytes, the application is able to set all the slaves to OP mode, and everything seems to be working fine.
Once adding the 7th slave I'm running into the same error. Only 1 LRW datagram is generated
[ 5568.457114] EtherCAT DEBUG 0: Adding datagram pair with expected WC 21.
[ 5568.457121] EtherCAT 0: Domain0: Logical address 0x00000000, 1659 byte, expected working counter 21.
[ 5568.466256] EtherCAT 0: Datagram domain0-0-main: Logical offset 0x00000000, 1659 byte, type LRW at f5ac490c.
The communication stops, probably because the master FSM is stuck in ec_master_send_datagrams()
1223 // does the current datagram fit in the frame?
1224 datagram_size = EC_DATAGRAM_HEADER_SIZE + datagram->data_size
1225 + EC_DATAGRAM_FOOTER_SIZE;
1226 if (cur_data - frame_data + datagram_size > ETH_DATA_LEN) {
1227 more_datagrams_waiting = 1;
1228 break;
1229 }
The code in ec_domain_finish() somehow misses the calculation of the PDO domain size, and never enters
if (fmmu->logical_domain_offset + fmmu->data_size - datagram_offset > EC_MAX_DATA_SIZE) {
And therefore creates a single LRW packet.
In dmesg, coming out from ec_domain_add_fmmu_config() I get
[ 5568.287037] EtherCAT DEBUG 0: Domain 0: Added 64 bytes at 0.
[ 5568.287418] EtherCAT DEBUG 0: Domain 0: Added 237 bytes at 0.
[ 5568.288753] EtherCAT DEBUG 0: Domain 0: Added 64 bytes at 237.
[ 5568.289188] EtherCAT DEBUG 0: Domain 0: Added 237 bytes at 237.
[ 5568.290699] EtherCAT DEBUG 0: Domain 0: Added 64 bytes at 474.
[ 5568.291140] EtherCAT DEBUG 0: Domain 0: Added 237 bytes at 474.
[ 5568.292549] EtherCAT DEBUG 0: Domain 0: Added 64 bytes at 711.
[ 5568.293007] EtherCAT DEBUG 0: Domain 0: Added 237 bytes at 711.
[ 5568.294599] EtherCAT DEBUG 0: Domain 0: Added 64 bytes at 948.
[ 5568.295069] EtherCAT DEBUG 0: Domain 0: Added 237 bytes at 948.
[ 5568.296778] EtherCAT DEBUG 0: Domain 0: Added 64 bytes at 1185.
[ 5568.297281] EtherCAT DEBUG 0: Domain 0: Added 237 bytes at 1185.
[ 5568.299003] EtherCAT DEBUG 0: Domain 0: Added 64 bytes at 1422.
[ 5568.299594] EtherCAT DEBUG 0: Domain 0: Added 237 bytes at 1422.
Later, printing out fmmu->data_size from ec_domain_finish() I get
ec_domain_finish(): fmmu->logical_domain_offset 0, fmmu->data_size=64, datagram_offset 0, valid_start 0, datagram_first_fmmu f18cdd24, valid_fmmu f18cdd24
ec_domain_finish(): fmmu->logical_domain_offset 237, fmmu->data_size=64, datagram_offset 0, valid_start 237, datagram_first_fmmu f18cdd24, valid_fmmu f18cd924
ec_domain_finish(): fmmu->logical_domain_offset 474, fmmu->data_size=64, datagram_offset 0, valid_start 474, datagram_first_fmmu f18cdd24, valid_fmmu f18cc124
ec_domain_finish(): fmmu->logical_domain_offset 711, fmmu->data_size=64, datagram_offset 0, valid_start 711, datagram_first_fmmu f18cdd24, valid_fmmu f18cc524
ec_domain_finish(): fmmu->logical_domain_offset 948, fmmu->data_size=64, datagram_offset 0, valid_start 948, datagram_first_fmmu f18cdd24, valid_fmmu f18cc924
ec_domain_finish(): fmmu->logical_domain_offset 1185, fmmu->data_size=64, datagram_offset 0, valid_start 1185, datagram_first_fmmu f18cdd24, valid_fmmu f18ccd24
ec_domain_finish(): fmmu->logical_domain_offset 1422, fmmu->data_size=64, datagram_offset 0, valid_start 1422, datagram_first_fmmu f18cdd24, valid_fmmu f18cd524
For some reason the fmmu's with data_size == 237 are not evaluated in the list_for_each_entry()
I will try to fix it myself, but I will gladly except any help offered.
Thanks,
Nir.
-----Original Message-----
From: Gavin Lambert <gavin.lambert at tomra.com>
Sent: Monday, April 26, 2021 2:16 AM
To: Geller, Nir <nir.geller at servotronix.com>; etherlab-users at etherlab.org
Subject: RE: [Etherlab-users] Running a large number of slaves
I assume you meant domain.c.
You are correct that such a configuration *should* work, provided that the slave actually supports it as well -- but it looks like you were just duplicating slave indexes rather than actually specifying unique objects, which is unlikely to work well. The SM/FMMU config on the master side has to exactly match the config on the slave side in order for communication to be successful.
You mention overlapping PDOs -- have you tried disabling that option? Certainly if your goal is to create a larger packet size then you should not have that enabled; and it's an experimental feature so it's more likely to have bugs in it, especially for unusual configurations such as what you were attempting.
Gavin Lambert
Senior Software Developer
COMPAC SORTING EQUIPMENT LTD | 4 Henderson Pl | Onehunga | Auckland 1061 | New Zealand
Switchboard: +49 2630 96520 | https://www.tomra.com
The information contained in this communication and any attachment is confidential and may be legally privileged. It should only be read by the person(s) to whom it is addressed. If you have received this communication in error, please notify the sender and delete the communication.
From: Geller, Nir <nir.geller at servotronix.com>
Sent: Monday, 26 April 2021 1:00 am
To: Gavin Lambert <gavin.lambert at tomra.com>; Richard Hacker <ha at igh.de>; etherlab-users at etherlab.org
Subject: RE: [Etherlab-users] Running a large number of slaves
Hi Gavin,
Thanks for your reply.
The final system I am developing will run about 80 slaves, each with a PDO domain size of ~130 bytes.
I do not have such a number of slaves at my disposal, so I'm trying to somehow simulate a PDO domain that exceeds 1486 bytes in order to test the ecat master.
I created 2 slaves, each with a PDO domain of about 920 bytes.
I also asked myself how the data will be split between the ethernet frames.
(Both slaves support overlapping PDOs).
I expected that 2 LRW datagrams will be created, each carrying the PDO data intended for a specific slave and sent in a separate ethernet frame. Why should this be a problem?
The situation is that only 1 LRW datagram is created, in the size of 1830 bytes.
This, of course, is not going to work.
In datagram.c, ec_domain_finish(),
The if statement
if (fmmu->logical_domain_offset + fmmu->data_size - datagram_offset > EC_MAX_DATA_SIZE) {
is never executed, but
/* Allocate last datagram pair, if data are left (this is also the case if
* the process data fit into a single datagram) */
if (domain->data_size > datagram_offset) {
Is executed only once.
Could you direct me more precisely where in the code a fix should be implemented?
Thanks,
Nir.
From: Gavin Lambert <mailto:gavin.lambert at tomra.com>
Sent: Friday, April 23, 2021 1:13 AM
To: Geller, Nir <mailto:nir.geller at servotronix.com>; Richard Hacker <mailto:ha at igh.de>; mailto:etherlab-users at etherlab.org
Subject: RE: [Etherlab-users] Running a large number of slaves
The EtherCAT bus in general is not designed for that kind of thing. When you have a large PDO domain, it is supposed to be because you have a large number of smaller slaves, not a small number of large slaves.
I'm not sure exactly where the cutoff should be, but most likely any single slave wanting to exchange more than 300 bytes or so should probably be using SDO rather than PDO. (SDO mailboxes can be configured up to 1484 bytes, although that depends on slave implementation too and most only support a much smaller size.) Do you really need that amount of data each and every cycle?
The packet-splitting does work - I have systems where each cycle sends three PDO packets - but the largest amount of PDO data in any one slave in my network is about 200 bytes. Most slaves are a lot less.
Gavin Lambert
Senior Software Developer
COMPAC SORTING EQUIPMENT LTD | 4 Henderson Pl | Onehunga | Auckland 1061 | New Zealand
Switchboard: +64 96 34 00 88 | https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.tomra.com%2F&data=04%7C01%7Cgavin.lambert%40tomra.com%7Cf585962eae3b49c600a908d907ea0f1d%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637549524141805369%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=0B1lreUtpyexzwDAC2%2BQOW3pJCU8BHURzAHJ%2F0niM3E%3D&reserved=0
The information contained in this communication and any attachment is confidential and may be legally privileged. It should only be read by the person(s) to whom it is addressed. If you have received this communication in error, please notify the sender and delete the communication.
From: Geller, Nir <mailto:nir.geller at servotronix.com>
Sent: Friday, 23 April 2021 1:39 am
To: Geller, Nir <mailto:nir.geller at servotronix.com>; Gavin Lambert <mailto:gavin.lambert at tomra.com>; Richard Hacker <mailto:ha at igh.de>; mailto:etherlab-users at etherlab.org
Subject: RE: [Etherlab-users] Running a large number of slaves
This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi,
In my setup I have 2 ecat slaves with a large PDO data worth of 917 bytes.
A LRW datagram is allocated with a payload size of 1830 bytes:
[ 56.206673] EtherCAT DEBUG 0: Adding datagram pair with expected WC 6.
[ 56.206690] EtherCAT 0: Domain0: Logical address 0x00000000, 1830 byte, expected working counter 6.
[ 56.215738] EtherCAT 0: Datagram domain0-0-main: Logical offset 0x00000000, 1830 byte, type LRW at f1c6600c.
My suspicion is that in master.c, ec_master_send_datagrams(), the following piece of code
// does the current datagram fit in the frame?
datagram_size = EC_DATAGRAM_HEADER_SIZE + datagram->data_size
+ EC_DATAGRAM_FOOTER_SIZE;
if (cur_data - frame_data + datagram_size > ETH_DATA_LEN) {
more_datagrams_waiting = 1;
break;
}
Gets stuck in an infinite loop because it can't handle a datagram larger than 1500 bytes.
Is my assumption correct?
Do you happen to have a code fix for this situation?
Thanks,
Nir.
From: Etherlab-users <mailto:etherlab-users-bounces at etherlab.org> On Behalf Of Geller, Nir
Sent: Wednesday, April 21, 2021 11:10 AM
To: Gavin Lambert <mailto:gavin.lambert at tomra.com>; Richard Hacker <mailto:ha at igh.de>; mailto:etherlab-users at etherlab.org
Subject: Re: [Etherlab-users] Running a large number of slaves
Hello again,
I tried running the ethercat master with a large PDO domain, but with no success.
Following
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsourceforge.net%2Fu%2Fuecasm%2Fetherlab-patches%2Fci%2Fdefault%2Ftree%2F%23readme&data=04%7C01%7Cgavin.lambert%40tomra.com%7Cf585962eae3b49c600a908d907ea0f1d%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637549524141805369%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MGM36BEcRcZajEQ0lVG2mBU6L%2B3N%2FYN7deptqxTsePk%3D&reserved=0
I built the ethercat master with Gavin's patch set.
I'm running on a x86 Intel Atom dual core with linux kernel 3.18.48. Ethernet adapter is igb.
To achieve a very large PDO volume I created 2 ecat slaves, each with PDO data worth of 917 bytes.
When connecting only 1 slave, and running examples/user/ec_user_example I can raise the slave to OP and exchange data between the master and slave over PDO.
When connecting 2 slaves the start up process of the ethercat master gets stuck after
EtherCAT DEBUG 0-main-0: Checking system time offset.
And according to a wireshark capture the communication completely stops even though the application is still running cyclically.
Can you please help me setup a functional system?
Thanks,
Nir.
________________________________________
From: Etherlab-users <mailto:etherlab-users-bounces at etherlab.org> on behalf of Geller, Nir <mailto:nir.geller at servotronix.com>
Sent: Wednesday, March 31, 2021 1:48 PM
To: Gavin Lambert <mailto:gavin.lambert at tomra.com>; Richard Hacker <mailto:ha at igh.de>; mailto:etherlab-users at etherlab.org <mailto:etherlab-users at etherlab.org>
Subject: Re: [Etherlab-users] Running a large number of slaves
Hi Gavin,
This sounds promising.
With regard to cyclic real time performance, does fragmentation work properly and efficiently with slaves that support DC?
Thanks,
Nir.
-----Original Message-----
From: Gavin Lambert <mailto:gavin.lambert at tomra.com>
Sent: Tuesday, March 30, 2021 9:40 AM
To: Geller, Nir <mailto:nir.geller at servotronix.com>; Richard Hacker <mailto:ha at igh.de>; mailto:etherlab-users at etherlab.org
Subject: RE: [Etherlab-users] Running a large number of slaves
Yes, it splits to multiple packets automatically. Just be careful to not use more data than your cycle rate will allow.
Note that initialization and configuration of a large number of slaves is very slow by default, as it occurs in series.
The unofficial patchset changes this to occur in parallel (for groups at a time rather than the whole network, to avoid creating too many packets at once).
Gavin Lambert
Senior Software Developer
COMPAC SORTING EQUIPMENT LTD | 4 Henderson Pl | Onehunga | Auckland 1061 | New Zealand
Switchboard: +49 2630 96520 | https://www.tomra.com
The information contained in this communication and any attachment is confidential and may be legally privileged. It should only be read by the person(s) to whom it is addressed. If you have received this communication in error, please notify the sender and delete the communication.
-----Original Message-----
From: Geller, Nir
Sent: Tuesday, 30 March 2021 1:23 am
To: Richard Hacker <mailto:ha at igh.de>; mailto:etherlab-users at etherlab.org
Subject: Re: [Etherlab-users] Running a large number of slaves
Hi,
Thanks for your reply.
You mean that in the case of a large amount of PDO data ( > 1500), a single invoke of ecrt_master_send(master) will result several frames sent out 1 after another?
Nir.
-----Original Message-----
From: Etherlab-users <mailto:etherlab-users-bounces at etherlab.org> On Behalf Of Richard Hacker
Sent: Monday, March 29, 2021 3:09 PM
To: mailto:etherlab-users at etherlab.org
Subject: Re: [Etherlab-users] Running a large number of slaves
EtherCAT and the master are not limited to the ethernet packet size.
EtherCAT frames are automatically divided into smaller ethernet packets as required. As long as you're not exceeding physical limits, (like sending ~1,5kb at a rate of 1kHz), you should be fine.
Physically EtherCAT can address ~64k slaves on a network.
On 2021-03-29 13:22, Geller, Nir wrote:
> Hi There,
>
> I'm trying to setup one ethercat master with a very large number of
> ethercat slaves.
>
> The first obstacle I'm thinking about is a very large amount of data
> sent over PDO each cycle, that will definitely exceed 1500 bytes.
>
> In order to address this issue I want to understand if it is possible
> to send more than one frame each cycle?
>
> Another method could be using jumbo frames. Does the ethercat master
> support that?
>
> Does anybody have practical experience with such a setup?
>
> Thanks,
>
> Nir.
>
>
Mit freundlichem Gruß
Richard Hacker
--
------------------------------------------------------------------------
Richard Hacker M.Sc.
mailto:richard.hacker at igh.de
Tel.: +49 201 / 36014-16
Ingenieurgemeinschaft IgH
Gesellschaft für Ingenieurleistungen mbH Nordsternstraße 66
D-45329 Essen
Amtsgericht Essen HRB 11500
USt-Id.-Nr.: DE 174 626 722
Geschäftsführung:
- Dr.-Ing. Siegfried Rotthäuser
- Dr. Sven Beermann, Prokurist
Tel.: +49 201 / 360-14-0
https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.igh.de%2F&data=04%7C01%7Cgavin.lambert%40tomra.com%7Cf585962eae3b49c600a908d907ea0f1d%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637549524141815365%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BNFstv%2FIi7753YpErDNWbQb7AvhJCB1dBVknH4AHgTA%3D&reserved=0
------------------------------------------------------------------------
--
Etherlab-users mailing list
mailto:Etherlab-users at etherlab.org
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.etherlab.org%2Fmailman%2Flistinfo%2Fetherlab-users&data=04%7C01%7Cgavin.lambert%40tomra.com%7Cf585962eae3b49c600a908d907ea0f1d%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637549524141825363%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=nfR4P9wUoE1emFmvFt%2BFwxVJ2gmMqCGH%2BYT4ZMxxdI0%3D&reserved=0
--
Etherlab-users mailing list
mailto:Etherlab-users at etherlab.org
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.etherlab.org%2Fmailman%2Flistinfo%2Fetherlab-users&data=04%7C01%7Cgavin.lambert%40tomra.com%7Cf585962eae3b49c600a908d907ea0f1d%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637549524141825363%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=nfR4P9wUoE1emFmvFt%2BFwxVJ2gmMqCGH%2BYT4ZMxxdI0%3D&reserved=0
--
Etherlab-users mailing list
mailto:Etherlab-users at etherlab.org
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.etherlab.org%2Fmailman%2Flistinfo%2Fetherlab-users&data=04%7C01%7Cgavin.lambert%40tomra.com%7Cf585962eae3b49c600a908d907ea0f1d%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C637549524141835360%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=HbTh9JjhWtoDnQINydq4Vp3oms8v2zTwpjG1DD0G0PE%3D&reserved=0
More information about the Etherlab-users
mailing list