Quote:
Originally Posted by rrossbach
I would STRONGLY encourage the folks handling the FMS to apply sound engineering and diagnostic principles to this problem.
|
We have a running QOS log on every switch in the network that can be accessed through a dedicated VPN network running back to the FMS engineering server, QOS records any error in communication that switches have, I did not find any FMS network errors at the events I was able to check over the weekend including NJ and NH. I have an engineering log running on every robot running during any match which logs every DS message recieved and records it along side the signals we recieve from the PLC IO, I have not found an instance yet where a DS dropped and the PLC IO errored out at the same time (PLC IO communicates every 20ms), still reviewing. The logs from the DSes that dropped out over the weekend during matches indicate that DSes stopped sending FMS status messages, FMS timed out listening for the messages and went into try to reacquire pinging to get the DSes reacquired. What this means is the DSes reset. It seems the DSes are resetting from the EMP that occurs when a robot that has a high voltage charged up from spinning its wheels out in the middle of the field gets in proximity of a field boundary (aluminum structure that is grounded) and does the EMP zap. Robots hitting the end of the field aren't resetting DSes from the physical impact, its the EMP discharge.
Looking into ways to mitigate the susceptability of the DSes to EMP because of the massive static issues that are occuring at some events. You'll notice the issue gets worse towards the end of a tournament because there's usually more static generated then from increased robot action in the elims.
I know its easy to blame the field because this was the first time you all were introduced to it, but I tried to account for these type issues in my designs. My background is semiconductor automation, so I know something about static sensitivities.
New complex system put together with hundreds of other complex systems for the first time, things to figure out, this is engineering...
We need to isolate the issues and then present the solutions. I put this out so you would know what I do at this time. Static is not our friend.
Edit: also, the network is gigabit, we're only utilizing a couple percent of it now, no traffic "congestion" issues.