[etherlab-dev] Missing Vendor ID / Product Code

[Graeme Foot]

Hi,

"/features/parallel-slave/0008-fsm_sii-loading-check.patch" is still required.

This patch fixes a problem where the SII data may still be loading from EEPROM while the slave fsm starts to read it, resulting in bad SII data.  This situation can occur when rescanning the slaves after a hotplug and one of the newly connected slaves is a little slow reading from the EEPROM (in my case EL1008 modules in particular).

It's probably worth being a base patch also.  I don't think there's anything relying on the parallel-slave patches.  It's just that it's more of an issue after the parallel-slave patches as you initialise more modules sooner so it's more likely to happen.

Cheers,
Graeme.


From: Gavin Lambert <gavin.lambert at tomra.com>
Sent: Tuesday, 11 June 2019 12:41 PM
To: Graeme Foot <Graeme.Foot at touchcut.com>; etherlab-dev at etherlab.org
Subject: RE: Missing Vendor ID / Product Code

Ah, I see.  I think the original version relied on simply not sending those datagrams later on.  Usually there’s only one or two cycles where a slave FSM is “blocked” like that, so it didn’t eat up too many datagrams and didn’t “wrap around” to cause issues with other slaves due to the limit on parallel FSMs.

But yes, you’re correct that it’s better to not consume the datagram from the ring in the first place, especially if there’s going to be prolonged inactivity.

I’ll probably end up putting this in the base patches, perhaps even folding it into one of the existing ones.  It’s probably getting about time to make another patchset release soon anyway. 😊

Does this supersede your “fsm_sii-loading-check.patch” or do you think that’s still useful?


Gavin Lambert
Senior Software Developer


[cid:image001.png at 01D52054.31E09080]
[TOMRA]<http://www.compacsort.com>[Facebook]<https://www.facebook.com/Compacsort>[Linkedin]<https://www.linkedin.com/company/compac-sorting-equipment/>[Youtube]<https://vimeo.com/compacsort>[twitter]<https://twitter.com/compacsort>

COMPAC SORTING EQUIPMENT LTD | 4 Henderson Pl | Onehunga | Auckland 1061 | New Zealand
Switchboard: +64 96 34 00 88 | tomra.com<http://www.tomra.com>

The information contained in this communication and any attachment is confidential and may be legally privileged. It should only be read by the person(s) to whom it is addressed. If you have received this communication in error, please notify the sender and delete the communication.

From: Graeme Foot <Graeme.Foot at touchcut.com<mailto:Graeme.Foot at touchcut.com>>
Sent: Tuesday, 11 June 2019 12:22
To: Gavin Lambert <gavin.lambert at tomra.com<mailto:gavin.lambert at tomra.com>>; etherlab-dev at etherlab.org<mailto:etherlab-dev at etherlab.org>
Subject: RE: Missing Vendor ID / Product Code

Hi,

Yes I saw "base/0019..." adding EC_DATAGRAM_INVALID.  I didn't explicitly see "base/0026..." adding the code to prevent abandoning the mailbox fsms but I was working with it's merged code.  "base/0026..." added another case where the external datagram queue has the issue due to not abandoning the fsm, but also not using the datagram.

My patch will plug that hole now by not incrementing the queue index if the datagram is not used (if flagged with EC_DATAGRAM_INVALID).

Regards,
Graeme.


From: Gavin Lambert <gavin.lambert at tomra.com<mailto:gavin.lambert at tomra.com>>
Sent: Tuesday, 11 June 2019 12:02 PM
To: Graeme Foot <Graeme.Foot at touchcut.com<mailto:Graeme.Foot at touchcut.com>>; etherlab-dev at etherlab.org<mailto:etherlab-dev at etherlab.org>
Subject: RE: Missing Vendor ID / Product Code

Did you have a look at base/0026-Prevent-abandoning-the-mailbox-state-machines-early-.patch?  Because that does something similar.

(It was base/0019-Support-for-multiple-mailbox-protocols.patch which added the handling of the INVALID datagram state for the mailbox state machines.  The one above was a bugfix for this patch, essentially.)


Gavin Lambert
Senior Software Developer


[cid:image001.png at 01D52054.31E09080]
[TOMRA]<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.compacsort.com&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cdd0195eb3ce54dbefac408d6ee02e76c%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C636958093607009911&sdata=pR5aviztMJhNMWW34T%2FbDK1RZtQToowI6GNjZJts93w%3D&reserved=0>[Facebook]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FCompacsort&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cdd0195eb3ce54dbefac408d6ee02e76c%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C636958093607019906&sdata=%2B7YFQ8sYT9OUbBED8vwo4eVy8t0bEfZGyn7Qfbm1pRI%3D&reserved=0>[Linkedin]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fcompac-sorting-equipment%2F&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cdd0195eb3ce54dbefac408d6ee02e76c%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C636958093607019906&sdata=V0GyMdX8DgYSfzZH81g%2Bg2vYXCCwuj%2FCwQMK02Lqj0k%3D&reserved=0>[Youtube]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvimeo.com%2Fcompacsort&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cdd0195eb3ce54dbefac408d6ee02e76c%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C636958093607029901&sdata=UoA4SQcWDw9TAFLP9A2MVKP4Bl%2B1tY19Kn6q16DdBDg%3D&reserved=0>[twitter]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fcompacsort&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cdd0195eb3ce54dbefac408d6ee02e76c%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C636958093607029901&sdata=W%2BOfvs51VEO%2Bqi9m%2B78eVLoY%2B2irh3MyoX3IWpqLXQM%3D&reserved=0>

COMPAC SORTING EQUIPMENT LTD | 4 Henderson Pl | Onehunga | Auckland 1061 | New Zealand
Switchboard: +64 96 34 00 88 | tomra.com<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.tomra.com&data=02%7C01%7Cgavin.lambert%40tomra.com%7Cdd0195eb3ce54dbefac408d6ee02e76c%7C4308d118edd143008a37cfeba8ad5898%7C0%7C0%7C636958093607029901&sdata=UDw5M9AhWocxlt0x1HZ2ddDs5NL5yAWaYNliVXQPtHY%3D&reserved=0>

The information contained in this communication and any attachment is confidential and may be legally privileged. It should only be read by the person(s) to whom it is addressed. If you have received this communication in error, please notify the sender and delete the communication.

From: Graeme Foot <Graeme.Foot at touchcut.com<mailto:Graeme.Foot at touchcut.com>>
Sent: Tuesday, 11 June 2019 11:52
To: etherlab-dev at etherlab.org<mailto:etherlab-dev at etherlab.org>
Cc: Gavin Lambert <gavin.lambert at tomra.com<mailto:gavin.lambert at tomra.com>>
Subject: RE: Missing Vendor ID / Product Code

Hi,

Unfortunately "0008-fsm_sii-loading-check.patch" (below) didn't fix my main problem.  It turns out it is an inherent problem with how the masters external datagram ring works.  I have attached a patch that plugs the hole causing the problem I was having but there may be other cases where issues could occur.

Patch: /features/parallel-slave/0009-ec_master_exec_slave_fsms-external-datagram-fix.patch


The guts of the problem:

ec_master_exec_slave_fsms() calls ec_master_get_external_datagram() to get a datagram from the external datagram ring.  The datagram is then passed to ec_fsm_slave_exec() of the slaves with some work to do.  This call will then return either 1 for fsm still in progress or 0 for fsm is complete.  The master assumes that if the fsm is still in progress then the datagram has been consumed and is in use, but there are various cases where this is not true.  If any of these cases occur then in the first loop of ec_master_exec_slave_fsms() these slaves fsm's may be executed multiple times while another slaves fsm is waiting on its datagram to return.

If too many slaves, or cycles, occur during this time then the waiting slaves datagram either gets its state set to EC_DATAGRAM_INVALID or gets reused by another slave.  This can lead to "cancelled" datagram replies or the two slaves getting the results from the second slaves datagram (as the first datagram index will be replaced and its reply is lost).


In my case this was occurring due to using the "0001-load-sii-from-file.patch" patch.  During the SII config stage of a slave this patch will create a kthread to attempt to read the SII file from disk.  In the meantime the ec_fsm_slave_exec() command will continue returning a value of 1 (fsm in progress) but will not be using the presented datagrams (setting the datagram state to EC_DATAGRAM_INVALID).

During initial startup and configuration of the master the ec_master_exec_slave_fsms() call is made from ec_master_idle_thread() in a loop with (in my configuration) a call to schedule() before resuming the loop.  This means that multiple loops may occur before a reply to a slaves datagram returns, leaving plenty of time for the in-use datagrams to be recycled resulting in its state or data being overwritten.


The patch I have attached now also tests the datagrams state for EC_DATAGRAM_INVALID before incrementing the external datagram ring index.  This solves my problem where the datagrams state is being set to EC_DATAGRAM_INVALID while waiting for the kthread to complete.

I suspect there may be other instances where this problem could occur.  One case I have thought of, but haven't been able to confirm, is when multiple protocols try to access a slaves mailbox at the same time (e.g. COE, EOE, FOE etc).  Only one protocol is allowed to communicate at a time.  The other protocols will be offered a datagram from the ring, but they aren't able to use it until their turn comes up.  In these cases if ec_read_mbox_locked() fails the datagram state is also set to EC_DATAGRAM_INVALID so the patch should also cover this case.


Regards,
Graeme.


From: etherlab-dev <etherlab-dev-bounces at etherlab.org<mailto:etherlab-dev-bounces at etherlab.org>> On Behalf Of Graeme Foot
Sent: Monday, 4 March 2019 2:36 PM
To: etherlab-dev at etherlab.org<mailto:etherlab-dev at etherlab.org>
Subject: Re: [etherlab-dev] Missing Vendor ID / Product Code

Hi,

I think I've finally solved the problem.  The slaves with the issue are returning with the "EEPROM not loaded" bit set when reading the SII information (bit 12 if the EEPROM status word).  If this bit is set then the slave has not yet finished reading the SII information from the EEPROM and the data returned may not be valid.  The master code was not checking for this bit.  I have attached a patch to do so:
/features/parallel-slave/0008-fsm_sii-loading-check.patch

The patch checks if the bit is set and keeps re-reading the EEPROM data until it is not.  At this point the data returned is still incorrect so a complete read is requested (where a write is first sent asking for the slave to load the data that needs to be read).  There is a 500ms timeout waiting for the bit to be clear.  If the bit does not clear then the EEPROM load may have failed (e.g. incorrect CRC value).


My previous patch (features/sii-read-failure/0001-sii-read-retry.patch) should no longer be required, but it may help to make reading of the SII data more robust.  I've attached the latest version of this one also.  It is now:
features/sii-read-failure/0001-slave-scan-retry.patch


Regards,
Graeme Foot.


From: etherlab-dev <etherlab-dev-bounces at etherlab.org<mailto:etherlab-dev-bounces at etherlab.org>> On Behalf Of Graeme Foot
Sent: Friday, 12 October 2018 11:38 AM
To: Gavin Lambert <gavin.lambert at tomra.com<mailto:gavin.lambert at tomra.com>>; etherlab-dev at etherlab.org<mailto:etherlab-dev at etherlab.org>
Subject: Re: [etherlab-dev] Missing Vendor ID / Product Code

Hi,

I've had a chance to play with my testrig and have managed to consistently reproduce the problem when hot-plugging a module (I haven't had the problem again on a production machine from a normal startup that I can test on yet).

My system:
- CX2020
- EK1110 (alias 10001)
- EK1100 (alias 20000)
- EL1008 (alias 1)

I start the system without the EL1008 plugged in and get it running.  I then plug in the EL1008 and the SII information fails to read, resulting in a zero alias, vendorID and product code etc.  I have attached a patch which resolves the issue on my testrig (but I don't know if it will resolve my production issue).

With --enable-sii-override set, the patch detects a zero vendorID or product code (hopefully no one has a device with a zero product code) and then retries scanning the slave after a 100ms timeout.  If --enable-sii-override is not set then it will do the retry if reading the SII size fails.

One strange thing I found during testing was that:
- When --enable-sii-override is not set ec_fsm_sii_success() would fail during the ec_fsm_slave_scan_state_sii_size() state; but
- When --enable-sii-override is set ec_fsm_sii_success() does not fail during the ec_fsm_slave_scan_state_sii_device() state, so I instead check for a zero vendorID or product code.

Gavin, the patch is against your previous patchset.  I put the patch under:
features/sii-read-failure/0001-sii-read-retry.patch

Let me know if you think there's anything dodgy with it.


Graeme.


From: Gavin Lambert <gavin.lambert at tomra.com<mailto:gavin.lambert at tomra.com>>
Sent: Wednesday, 8 August 2018 3:13 PM
To: Graeme Foot <Graeme.Foot at touchcut.com<mailto:Graeme.Foot at touchcut.com>>; etherlab-users at etherlab.org<mailto:etherlab-users at etherlab.org>
Subject: RE: Missing Vendor ID / Product Code

There’s lots of things that can cause that.  Most often, I’ve seen this when packets get lost or corrupted, so the initial discovery datagrams get lost or fail.  Usually bad wiring or shielding is the culprit.

I think it might be possible to get something similar due to an unfortunate timing coincidence – if the devices are being connected “live” then a dodgy plug-in could make the device visible in the initial device count scan, but then disconnected before it finishes the identity discovery, but then reconnected again before it does the next device count scan (so it doesn’t try again).  Replugging the devices (with less unfortunate timing) or restarting the etherlab service should both recover from that case, however.

Or, of course, you might have found a bug. 😊

It's hard to say for sure what actually happened without seeing syslogs and/or reproducing it.

From: Graeme Foot
Sent: Wednesday, 8 August 2018 14:12
To: etherlab-users at etherlab.org<mailto:etherlab-users at etherlab.org>
Subject: [etherlab-users] Missing Vendor ID / Product Code

Hi,

I updated my EtherCAT system to use Gavin's patch set (revision 10, 20171108).  It has been running fine on a few machines, but have just had a machine being commissioned where one of the slave modules had a zero Vendor ID and Product Code (and I suspect it failed to read any information from the slave).  Unfortunately it occurred while I was not available so our engineers reverted to the previous version (which detected the module correctly) and shipped the machine, so I have very minimal information and no logs.

The module with the problem was the 17th module, the first EL2612 of 5.  It is directly after an EL9410 power module.  It has an explicit alias set.  The engineers had tried repowering the whole system and replacing the module.

Until I get a machine to test on with the same behaviour I was wondering if anyone else has had problems with slaves not initialising correctly.

Thanks,
Graeme Foot.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.etherlab.org/pipermail/etherlab-dev/attachments/20190611/11b87cb8/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 11438 bytes
Desc: image001.png
URL: <http://lists.etherlab.org/pipermail/etherlab-dev/attachments/20190611/11b87cb8/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 1629 bytes
Desc: image002.png
URL: <http://lists.etherlab.org/pipermail/etherlab-dev/attachments/20190611/11b87cb8/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 1750 bytes
Desc: image003.png
URL: <http://lists.etherlab.org/pipermail/etherlab-dev/attachments/20190611/11b87cb8/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 1855 bytes
Desc: image004.png
URL: <http://lists.etherlab.org/pipermail/etherlab-dev/attachments/20190611/11b87cb8/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 1970 bytes
Desc: image005.png
URL: <http://lists.etherlab.org/pipermail/etherlab-dev/attachments/20190611/11b87cb8/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 20278 bytes
Desc: image006.png
URL: <http://lists.etherlab.org/pipermail/etherlab-dev/attachments/20190611/11b87cb8/attachment-0011.png>