[etherlab-dev] Missing Vendor ID / Product Code

[Graeme Foot]

Hi,

Unfortunately "0008-fsm_sii-loading-check.patch" (below) didn't fix my main problem.  It turns out it is an inherent problem with how the masters external datagram ring works.  I have attached a patch that plugs the hole causing the problem I was having but there may be other cases where issues could occur.

Patch: /features/parallel-slave/0009-ec_master_exec_slave_fsms-external-datagram-fix.patch


The guts of the problem:

ec_master_exec_slave_fsms() calls ec_master_get_external_datagram() to get a datagram from the external datagram ring.  The datagram is then passed to ec_fsm_slave_exec() of the slaves with some work to do.  This call will then return either 1 for fsm still in progress or 0 for fsm is complete.  The master assumes that if the fsm is still in progress then the datagram has been consumed and is in use, but there are various cases where this is not true.  If any of these cases occur then in the first loop of ec_master_exec_slave_fsms() these slaves fsm's may be executed multiple times while another slaves fsm is waiting on its datagram to return.

If too many slaves, or cycles, occur during this time then the waiting slaves datagram either gets its state set to EC_DATAGRAM_INVALID or gets reused by another slave.  This can lead to "cancelled" datagram replies or the two slaves getting the results from the second slaves datagram (as the first datagram index will be replaced and its reply is lost).


In my case this was occurring due to using the "0001-load-sii-from-file.patch" patch.  During the SII config stage of a slave this patch will create a kthread to attempt to read the SII file from disk.  In the meantime the ec_fsm_slave_exec() command will continue returning a value of 1 (fsm in progress) but will not be using the presented datagrams (setting the datagram state to EC_DATAGRAM_INVALID).

During initial startup and configuration of the master the ec_master_exec_slave_fsms() call is made from ec_master_idle_thread() in a loop with (in my configuration) a call to schedule() before resuming the loop.  This means that multiple loops may occur before a reply to a slaves datagram returns, leaving plenty of time for the in-use datagrams to be recycled resulting in its state or data being overwritten.


The patch I have attached now also tests the datagrams state for EC_DATAGRAM_INVALID before incrementing the external datagram ring index.  This solves my problem where the datagrams state is being set to EC_DATAGRAM_INVALID while waiting for the kthread to complete.

I suspect there may be other instances where this problem could occur.  One case I have thought of, but haven't been able to confirm, is when multiple protocols try to access a slaves mailbox at the same time (e.g. COE, EOE, FOE etc).  Only one protocol is allowed to communicate at a time.  The other protocols will be offered a datagram from the ring, but they aren't able to use it until their turn comes up.  In these cases if ec_read_mbox_locked() fails the datagram state is also set to EC_DATAGRAM_INVALID so the patch should also cover this case.


Regards,
Graeme.


From: etherlab-dev <etherlab-dev-bounces at etherlab.org> On Behalf Of Graeme Foot
Sent: Monday, 4 March 2019 2:36 PM
To: etherlab-dev at etherlab.org
Subject: Re: [etherlab-dev] Missing Vendor ID / Product Code

Hi,

I think I've finally solved the problem.  The slaves with the issue are returning with the "EEPROM not loaded" bit set when reading the SII information (bit 12 if the EEPROM status word).  If this bit is set then the slave has not yet finished reading the SII information from the EEPROM and the data returned may not be valid.  The master code was not checking for this bit.  I have attached a patch to do so:
/features/parallel-slave/0008-fsm_sii-loading-check.patch

The patch checks if the bit is set and keeps re-reading the EEPROM data until it is not.  At this point the data returned is still incorrect so a complete read is requested (where a write is first sent asking for the slave to load the data that needs to be read).  There is a 500ms timeout waiting for the bit to be clear.  If the bit does not clear then the EEPROM load may have failed (e.g. incorrect CRC value).


My previous patch (features/sii-read-failure/0001-sii-read-retry.patch) should no longer be required, but it may help to make reading of the SII data more robust.  I've attached the latest version of this one also.  It is now:
features/sii-read-failure/0001-slave-scan-retry.patch


Regards,
Graeme Foot.


From: etherlab-dev <etherlab-dev-bounces at etherlab.org<mailto:etherlab-dev-bounces at etherlab.org>> On Behalf Of Graeme Foot
Sent: Friday, 12 October 2018 11:38 AM
To: Gavin Lambert <gavin.lambert at tomra.com<mailto:gavin.lambert at tomra.com>>; etherlab-dev at etherlab.org<mailto:etherlab-dev at etherlab.org>
Subject: Re: [etherlab-dev] Missing Vendor ID / Product Code

Hi,

I've had a chance to play with my testrig and have managed to consistently reproduce the problem when hot-plugging a module (I haven't had the problem again on a production machine from a normal startup that I can test on yet).

My system:
- CX2020
- EK1110 (alias 10001)
- EK1100 (alias 20000)
- EL1008 (alias 1)

I start the system without the EL1008 plugged in and get it running.  I then plug in the EL1008 and the SII information fails to read, resulting in a zero alias, vendorID and product code etc.  I have attached a patch which resolves the issue on my testrig (but I don't know if it will resolve my production issue).

With --enable-sii-override set, the patch detects a zero vendorID or product code (hopefully no one has a device with a zero product code) and then retries scanning the slave after a 100ms timeout.  If --enable-sii-override is not set then it will do the retry if reading the SII size fails.

One strange thing I found during testing was that:
- When --enable-sii-override is not set ec_fsm_sii_success() would fail during the ec_fsm_slave_scan_state_sii_size() state; but
- When --enable-sii-override is set ec_fsm_sii_success() does not fail during the ec_fsm_slave_scan_state_sii_device() state, so I instead check for a zero vendorID or product code.

Gavin, the patch is against your previous patchset.  I put the patch under:
features/sii-read-failure/0001-sii-read-retry.patch

Let me know if you think there's anything dodgy with it.


Graeme.


From: Gavin Lambert <gavin.lambert at tomra.com<mailto:gavin.lambert at tomra.com>>
Sent: Wednesday, 8 August 2018 3:13 PM
To: Graeme Foot <Graeme.Foot at touchcut.com<mailto:Graeme.Foot at touchcut.com>>; etherlab-users at etherlab.org<mailto:etherlab-users at etherlab.org>
Subject: RE: Missing Vendor ID / Product Code

There’s lots of things that can cause that.  Most often, I’ve seen this when packets get lost or corrupted, so the initial discovery datagrams get lost or fail.  Usually bad wiring or shielding is the culprit.

I think it might be possible to get something similar due to an unfortunate timing coincidence – if the devices are being connected “live” then a dodgy plug-in could make the device visible in the initial device count scan, but then disconnected before it finishes the identity discovery, but then reconnected again before it does the next device count scan (so it doesn’t try again).  Replugging the devices (with less unfortunate timing) or restarting the etherlab service should both recover from that case, however.

Or, of course, you might have found a bug. 😊

It's hard to say for sure what actually happened without seeing syslogs and/or reproducing it.

From: Graeme Foot
Sent: Wednesday, 8 August 2018 14:12
To: etherlab-users at etherlab.org<mailto:etherlab-users at etherlab.org>
Subject: [etherlab-users] Missing Vendor ID / Product Code

Hi,

I updated my EtherCAT system to use Gavin's patch set (revision 10, 20171108).  It has been running fine on a few machines, but have just had a machine being commissioned where one of the slave modules had a zero Vendor ID and Product Code (and I suspect it failed to read any information from the slave).  Unfortunately it occurred while I was not available so our engineers reverted to the previous version (which detected the module correctly) and shipped the machine, so I have very minimal information and no logs.

The module with the problem was the 17th module, the first EL2612 of 5.  It is directly after an EL9410 power module.  It has an explicit alias set.  The engineers had tried repowering the whole system and replacing the module.

Until I get a machine to test on with the same behaviour I was wondering if anyone else has had problems with slaves not initialising correctly.

Thanks,
Graeme Foot.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.etherlab.org/pipermail/etherlab-dev/attachments/20190610/a2ce1663/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0009-ec_master_exec_slave_fsms-external-datagram-fix.patch
Type: application/octet-stream
Size: 2266 bytes
Desc: 0009-ec_master_exec_slave_fsms-external-datagram-fix.patch
URL: <http://lists.etherlab.org/pipermail/etherlab-dev/attachments/20190610/a2ce1663/attachment-0003.obj>