Hi again,<br><br>That certainly shed some light on the situation. We finally came to the conclusion that we would split each thread's data in the same domain, even if they are working at the same rate.<br><br>(This email is quite long, so I divided it in sections that you could respond to without having to read to the end of the email)<br>
<br>___________________________________________________________________________<br><br>Let me explain to you our situation and then ask my question about master_send and master_receive.<br><br>We are working on a huge network of sensors and (not that many) actuators. By huge I mean about 30,000 pdo entries, so let's say 100 slaves. We didn't want to create many domains as that introduced considerable delay, but that seems inevitable.<br>
<br>However, the reason for splitting data in different threads is not the amount of data. The reason is first, the sensors provide data at different rates and therefore they are in different domains and are read at different rates (hence different threads with different cyclic times). There is a more important reason though. We need to have <b>high reliability</b>. In fact, the different sensor types all sense the same physical value and are overlayed so if one type fails, the other type provides that value.<br>
<br>Consequently, we cannot rely on <b>one</b> thread to do the job. That is why, for each domain, we have one thread so if one thread fails, the system is only partially down.<br><br>With the same argument, <b>we cannot rely on only one thread calling master_send and master_receive</b>. That is why I was trying to figure out a way to prevent unnecessary send and receives while each thread independently tries sending and receiving. (I wonder why you didn't answer this question though: <u>Is there a way to understand whether a sent frame has arrived back to master yet or not?</u>)<br>
<br>___________________________________________________________________________<br><br>Now we were wondering about one thing. If each thread, independently calls master_send and master_receive, what happens? Here are the situations that may arise that might cause problem and I would like to know if they are properly handled in EtherLab.<br>
<br>1. Thread 1 queues domain and sends the frame.<br> Thread 2 immediately after queues another domain and sends the frame.<br><br>In this situation, the two frames would be traversing the network one right after the other. Can the slaves handle that? Do they have queues for many frames arriving faster than they can process and forward them? Or does the master know that it shouldn't send the frames too fast?<br>
<br>2. Thread 1 queues domain and sends the frame<br> Thread 2 queues domain and sends the frame<br> In the network, both frames finish the cycle and return back to the master<br> Thread 1 wakes up and receives (then processes the domain)<br>
Thread 2 wakes up and receives (then processes the domain)<br><br>In this situation, does master_receive called by Thread 1 exchange data arrived from both frames or only one? In the former case, the call to master_receive by Thread 2 would observe that there are no new packets in the ethernet card. Can it handle that or does it assume there would always be a packet there?<br>
<br>3. Thread 1 queues domain and sends the frame<br> Thread 2 queues domain and sends the frame<br> Thread 1 dies!<br> Thread 2 wakes up and receives (then processes the domain)<br><br>This would somehow be answered by the answer to the previous situation but I just wanted to emphasize. In such a case, would there be a residue packet in the ethernet card (because Thread 2's master_receive call took only one frame) or would it properly exchange data from both arriving frames?<br>
<br>___________________________________________________________________________<br><br>Let me emphasize again why we can't have master_send and master_receive only in one thread (the fastest thread). One reason is reliability. If the fastest thread dies, we don't want the program to halt. The second reason is delay. Imagine these two threads:<br>
<br>Thread 1 working at period 40ms<br>Thread 2 working at period 30ms<br><br>According to you, we should have Thread 2 do the master_send and master_receive. Now consider this scenario:<br><br>1. Thread 1 queues domain and sends frame<br>
2. Immediately after Thread 2 queues another domain (but will not be sent, because Thread 2 came too late)<br>3. 30ms after (30ms after step 1), Thread 1 wakes up, exchanges data, calculates something, queues domain and sends frame (This time both domains are included in the frame)<br>
4. 10ms after (40ms after step 2), Thread 2 wakes up, but there is no new data. It queues its domain again and sleeps<br>5. 20ms after (30ms after step 3), Thread 1 wakes up, exchanges data etc<br>6. 20ms after (40ms after step 4), Thread 2 wakes up and finally gets the new data for its domain<br>
<br>As you can see, it took Thread 2, 80ms to get its data, which is twice as its period. The delay could have been reduced to a few milliseconds, even hundreds of microseconds if such a thing was done:<br><br>Each thread queues domain and sends<br>
loop while data has not arrived<br> sleep in the loop so it's not really busy waiting<br>The thread receives and processes<br><br>According to our measurements and calculations, this value can be less than 4ms for the huge network I mentioned in the beginning. With your suggestion we would be having 20 times the delay we could have had (only if it was possible to check if there is new packet arrived in the network card (Is it possible?))<br>
<br>___________________________________________________________________________<br><br>I really appreciate your time and effort and hope our use of EtherLab would also provide useful feedback for you,<br>Shahbaz<br><br><br>
<div class="gmail_quote">On Fri, Oct 14, 2011 at 5:11 PM, Richard Hacker <span dir="ltr"><<a href="mailto:ha@igh-essen.com">ha@igh-essen.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hello<br>
<br>
I think you do not quite understand what happens when:<br>
<br>
EtherCAT has 4 essential functions in cyclic mode:<br>
ecrt_master_receive(master_ptr): Fetches ethernet (yes etherNET!) data from<br>
the card. This ethernet packet contains all your input domains.<br>
ecrt_domain_process(domain_ptr): Processes the ethernet packet for domain<br>
ecrt_domain_queue(domain_ptr): Puts domain in a linked list to be sent<br>
ecrt_master_send(master_ptr): Transfers an ethernet packet made up of your<br>
input and output domains to the card<br>
<br>
Only the fastest thread should handle the pair ecrt_master_receive() and<br>
ecrt_master_send()! Different threads each handle the pair<br>
ecrt_domain_process() and ecrt_domain_queue(). Note: you must ensure that only<br>
one thread calls ecrt_domain_process() and ecrt_domain_queue() - protect these<br>
functions with semaphores.<br>
<br>
You should not have two threads running on the same domain. Open new domains<br>
for this purpose, after all, that is what domains are for!<br>
<br>
I cannot figure out from your description whether your threads have different<br>
frequencies. Having two threads running at the same rate will present you with<br>
some trouble, aka waking up or queueing just a little too late or too early.<br>
<br>
If you have so much data that you want to split things up into different<br>
threads, you sould consider using 2 masters.<br>
<font color="#888888"><br>
- Richard<br>
</font><div><div></div><div class="h5"><br>
On Friday 14 October 2011 15:54:04 Shahbaz Yousefi wrote:<br>
> Richard,<br>
><br>
> Thanks for the prompt reply. I understood what you mean and I understood<br>
> where my idea was wrong. There is however a question that still needs to be<br>
> answered.<br>
><br>
> Imagine you have one domain that contains two types of pdo entries. You<br>
> have two threads that each work on this same domain although on different<br>
> sections of it. (The reason they are two in one domain is that (if I<br>
> understood correctly) having more domains means more overhead in the<br>
> network, so we are grouping entries that have the same data rate in one<br>
> domain)<br>
><br>
> So, the threads go: calculate; exchange I/O; wait (therefore, get input;<br>
> calculate; write output; wait)<br>
><br>
> Now, imagine this sequence:<br>
> 1. Thread 1 finishes calculation and writes output<br>
> 2. Slightly after, Thread 2 finishes calculation and writes output on the<br>
> SAME domain<br>
> 3. After Thread1 wakes up, it receives the input<br>
> 4. Slightly after, Thread 2 wakes up and wants to receive input from the<br>
> SAME domain.<br>
><br>
> First question is, would this work with EtherLab? (So, does EtherLab itself<br>
> understand that at step 2, the exchange data of this domain is in progress<br>
> and would automatically ignore this step?)<br>
><br>
> I am assuming this won't work. Even if it does though, a solution such as<br>
> the following makes more sense to me:<br>
><br>
> function: ethercat manager - write output<br>
> if domain data exchange in progress<br>
> ignore (because data is already being exchanged)<br>
> else<br>
> domain_queue<br>
> master_send<br>
><br>
> function: ethercat manager - read input<br>
> if last write already read<br>
> ignore (and use the latest available data)<br>
> else if domain data exchange in progress<br>
> ignore (if data exchange in progress, it means the other thread has<br>
> issued data exchange recently)<br>
> else<br>
> master_receive<br>
> domain_process<br>
><br>
> Now each thread goes: ethercat manager - read input; calculate; ethercat<br>
> manager - write output; wait<br>
><br>
> This way the previous scenario would be like this:<br>
> 1. Thread 1 finishes calculation and writes output<br>
> 2. Slightly after, Thread 2 finishes calculation and write output is<br>
> ignored by the ethercat manager<br>
> 3. After Thread1 wakes up, it receives the input<br>
> 4. Slightly after, Thread 2 wakes up and uses the same results<br>
><br>
> Now, however another problem arises. What if Thread 1's read is ignored<br>
> because data is still being exchanged, and then while Thread 1 is<br>
> performing "calculate", Thread 2 performs a successful read and changes<br>
> the data, ruining Thread 1's work?<br>
><br>
> The second question is, regardless of the solution of the mentioned<br>
> problem, how do you check "domain data exchange is in progress" to<br>
> implement the ethercat manager functions in the first place? (That is,<br>
> when you issue a write output, how can you be sure when the read input is<br>
> issued, the data HAS actually finished exchanging?)<br>
><br>
> Thank you very much for your attention,<br>
> Shahbaz<br>
><br>
> P.S. The delay of one cycle is not a problem. That had been a<br>
> misunderstanding on my side.<br>
><br>
> P.S.2. I have been considering separating the pdo entries in more domains<br>
> so that different threads won't share domains. That however, would be the<br>
> last resort. If it is possible to have two threads working with the same<br>
> domain, I would be happier with that solution. If it is impossible, tell<br>
> me and I'll simply make sure no two threads use data from the same domain.<br>
><br>
> On Fri, Oct 14, 2011 at 10:01 AM, Richard Hacker <<a href="mailto:ha@igh-essen.com">ha@igh-essen.com</a>> wrote:<br>
> > Hi,<br>
> ><br>
> > I am not sure why you want to go through all this trouble. Of coarse, if<br>
> > your<br>
> > calculation is so long that there is no time to exchange IO, you have<br>
> > trouble<br>
> > looming anyway!<br>
> ><br>
> > So what do you want to do with the data if you receive it in the same<br>
> > cycle instead of waiting till your next call? For me, there is no point<br>
> > of busily waiting till your packet arrives, instead of being relaxed and<br>
> > receiving the<br>
> > packet next cycle.<br>
> ><br>
> > The normal run of a control program is:<br>
> > calculate; exchange io; wait; calculate; exchange io; wait; etc.<br>
> > where exchange io means: write output and get new inputs. Master receive<br>
> > and<br>
> > domain process simply fetches and processes the data that was transmitted<br>
> > with<br>
> > master_send at the end of you pseudo code examples.<br>
> ><br>
> > Now, exchange io is done in the background by the network card. This<br>
> > means, that you could call exchange io right at the start of your cycle<br>
> > and subsequently calculate. In this case your calculation and exchange io<br>
> > runs in<br>
> > parallel. This is useful when your calculation is long and you have a lot<br>
> > of<br>
> > data to exchange, i.e.<br>
> > exchange io; calculate; wait; exchange io; calculate wait; etc.<br>
> ><br>
> > The drawback is that your propagation time from input change to output<br>
> > reaction is 2 cycles, instead of only 1. That is the price to pay if you<br>
> > have<br>
> > lots of data and a long calculation - there is no free lunch!!!<br>
> ><br>
> > I do not think that you have a problem. Draw your operations on a time<br>
> > line and convince yourself that once you are in in the loop, you have max<br>
> > 1 cycle<br>
> > delay from input to output. If that is too long for you, then ethercat is<br>
> > not<br>
> > your solution. Then you need direct IO like that of microcontrollers and<br>
> > the<br>
> > like.<br>
> ><br>
> > - Richard<br>
> ><br>
> > On Thursday 13 October 2011 17:12:46 Shahbaz Yousefi wrote:<br>
> > > Hi,<br>
> > ><br>
> > > I have been working with etherlab recently and got ethercat working up<br>
> ><br>
> > and<br>
> ><br>
> > > everything is fine.<br>
> > ><br>
> > > There is a delay issue however that I'm concerned about. As seen in the<br>
> > > examples, the way you read from the network is like this (imagine you<br>
> > > are interested in reading sensor values):<br>
> > ><br>
> > > while (running)<br>
> > > {<br>
> > > master receive<br>
> > > domain process<br>
> > ><br>
> > > read data<br>
> > ><br>
> > > domain queue<br>
> > > master send<br>
> > ><br>
> > > rt wait period<br>
> > > }<br>
> > ><br>
> > > in which case you assume that the task period is long enough to be sure<br>
> > > that the packet sent in the bottom of the loop has returned when the<br>
> ><br>
> > loop<br>
> ><br>
> > > starts again and so you can receive the data.<br>
> > ><br>
> > > However, I was wondering if it is possible to, instead of taking an<br>
> > > upper bound of the time, simply check to see whether the data has<br>
> > > arrived or<br>
> ><br>
> > not.<br>
> ><br>
> > > After some research, I got to this code:<br>
> > ><br>
> > > while (running)<br>
> > > {<br>
> > > domain queue<br>
> > > master send<br>
> > ><br>
> > > do<br>
> > > {<br>
> > > sleep a little<br>
> > ><br>
> > > master receive<br>
> > > domain process<br>
> > ><br>
> > > ecrt_domain_state(domain, &state);<br>
> > ><br>
> > > } while (state.wc_state != EC_WC_COMPLETE && !timeout)<br>
> > ><br>
> > > read data<br>
> > ><br>
> > > rt wait period<br>
> > > }<br>
> > ><br>
> > > This way, after sending the packet, you would read the data as soon as<br>
> ><br>
> > they<br>
> ><br>
> > > arrive.<br>
> > ><br>
> > > The problem with this was that, besides the fact that early calls to<br>
> > > master_receive (or domain_process) generated a huge amount of warning<br>
> ><br>
> > about<br>
> ><br>
> > > working counter changing to 0/1 and back to 1/1 again, the kernel<br>
> > > started to at some point keep crashing.<br>
> > ><br>
> > > I would like to know, how can I detect when the packet has arrived<br>
> ><br>
> > without<br>
> ><br>
> > > knowing an upper bound about it and wait blindly by that much?<br>
> > ><br>
> > > Note: This is most useful for me for this reason:<br>
> > ><br>
> > > I may have different threads requesting data from a domain which<br>
> > > includes different sensors. Each type of sensor produces data at a<br>
> > > different rate and I would like to read the data at different rates to.<br>
> > > I don't want to (and I don't think is even possible) to have different<br>
> > > threads<br>
> ><br>
> > requesting<br>
> ><br>
> > > data from the same domain (because they may send the packet while the<br>
> ><br>
> > one<br>
> ><br>
> > > for the previous thread hasn't yet arrived). So what I want to is<br>
> > > this:<br>
> > ><br>
> > > ethercat coordinator:<br>
> > ><br>
> > > domain_updating = no<br>
> > ><br>
> > > send_request_for_domain<br>
> > > {<br>
> > > if (domain_updating == no)<br>
> > > {<br>
> > > domain_updating = yes<br>
> > > domain queue<br>
> > > master send<br>
> > > }<br>
> > > // else, do nothing, it is being updated!<br>
> > > }<br>
> > ><br>
> > > receive_from_domain()<br>
> > > {<br>
> > > while (ecrt_domain_data_not_yet_received) // this is the function I<br>
> ><br>
> > need<br>
> ><br>
> > > wait<br>
> > > domain_updating = no<br>
> > > // data available<br>
> > > }<br>
> > ><br>
> > > and then, each thread that wants something from the domain would look<br>
> ><br>
> > like<br>
> ><br>
> > > this:<br>
> > ><br>
> > > thread:<br>
> > > send_request_for_domain<br>
> > > receive_from_domain<br>
> > > read data<br>
> > ><br>
> > > This way, if two threads call send_request_for_domain at the same time,<br>
> > > only one of them would actually do the<br>
> > ><br>
> > > domain queue<br>
> > > master send<br>
> > ><br>
> > > and both of them use the result.<br>
> > ><br>
> > > I would appreciate it if you could shed some light on this matter.<br>
> > > Shahbaz<br>
> ><br>
> > Mit freundlichem Gruß<br>
> ><br>
> > Richard Hacker<br>
> ><br>
> > --<br>
> > ------------------------------------------------------------------------<br>
> ><br>
> > Richard Hacker M.Sc.<br>
> > <a href="mailto:richard.hacker@igh-essen.com">richard.hacker@igh-essen.com</a><br>
> > Tel.: <a href="tel:%2B49%20201%20%2F%2036014-16" value="+492013601416">+49 201 / 36014-16</a><br>
> ><br>
> > Ingenieurgemeinschaft IgH<br>
> > Gesellschaft für Ingenieurleistungen mbH<br>
> > Heinz-Bäcker-Str. 34<br>
> > D-45356 Essen<br>
> > Amtsgericht Essen HRB 11500<br>
> > USt-Id.-Nr.: DE 174 626 722<br>
> > Geschäftsführung:<br>
> > - Dr.-Ing. S. Rotthäuser,<br>
> > - Dr.-Ing. T. Finke,<br>
> > - Dr.-Ing. W. Hagemeister<br>
> > Tel.: <a href="tel:%2B49%20201%20%2F%20360-14-0" value="+49201360140">+49 201 / 360-14-0</a><br>
> > <a href="http://www.igh-essen.com" target="_blank">http://www.igh-essen.com</a><br>
> ><br>
> > ------------------------------------------------------------------------<br>
><br>
<br>
Mit freundlichem Gruß<br>
<br>
Richard Hacker<br>
<br>
--<br>
------------------------------------------------------------------------<br>
<br>
Richard Hacker M.Sc.<br>
<a href="mailto:richard.hacker@igh-essen.com">richard.hacker@igh-essen.com</a><br>
Tel.: <a href="tel:%2B49%20201%20%2F%2036014-16" value="+492013601416">+49 201 / 36014-16</a><br>
<br>
Ingenieurgemeinschaft IgH<br>
Gesellschaft für Ingenieurleistungen mbH<br>
Heinz-Bäcker-Str. 34<br>
D-45356 Essen<br>
Amtsgericht Essen HRB 11500<br>
USt-Id.-Nr.: DE 174 626 722<br>
Geschäftsführung:<br>
- Dr.-Ing. S. Rotthäuser,<br>
- Dr.-Ing. T. Finke,<br>
- Dr.-Ing. W. Hagemeister<br>
Tel.: <a href="tel:%2B49%20201%20%2F%20360-14-0" value="+49201360140">+49 201 / 360-14-0</a><br>
<a href="http://www.igh-essen.com" target="_blank">http://www.igh-essen.com</a><br>
<br>
------------------------------------------------------------------------<br>
</div></div></blockquote></div><br>