Team Fusion #364, Bayou Regional, FMS Woes

[Gary Dillard]

SPAM had dropped coms issues almost every match at the Orlando regional, they changed out 2 PD boards which were indicating some kind of loss of ground and it still didn't fix the problem. I don't know all the details - Eric was texting me during the competition - but it's starting to smell awfully fishy.

[RyanN]

Originally Posted by Gary Dillard

SPAM had dropped coms issues almost every match at the Orlando regional, they changed out 2 PD boards which were indicating some kind of loss of ground and it still didn't fix the problem. I don't know all the details - Eric was texting me during the competition - but it's starting to smell awfully fishy.

Here's my stance on it the PD board. Besides it making no sense... how is it that a consumer grade Linksys router (the one provided on the practice field) is able to keep a connection, while FIRST's field system using high quality equipment cannot keep a connection if there is really ripple from the PD board, even while still?

If we do get to Lone Star, we will be swapping out the PD board, because that is the last thing to swap. I'm also loading our stripped down code and going bare minimal for at least our first match for Thursday to verify we're all good to go, then increase the complexity throughout the day.

[RufflesRidge]

Ryan, any chance you can post the DS logs (they will be chopped on each disconnect) from any of the matches where you has issues? I'm curious how they compare to the ones that Joe posted from 330.

From what I saw on the webcast you guys did move a bit in one of your matches on Saturday morning, that would be a good one to get the logs from if possible.

[Jared Russell]

When you guys run wirelessly on the practice field, are you using WPA/WEP? That is the only potential difference between practice and FMS on the robot side. Perhaps the regulated 12V supply to the wireless bridge is affecting encrypted operations...?

I know the radio configuration software has changed since last season, and I suspect on average robots are far more bandwidth hungry than in the past (lots of cameras streaming back to Dashboards). Those might be two other factors causing the failure when other robots are active?

What a heartbreaking thread to read.

[wireties]

Originally Posted by RyanN

Here's my stance on it the PD board. Besides it making no sense... how is it that a consumer grade Linksys router (the one provided on the practice field) is able to keep a connection, while FIRST's field system using high quality equipment cannot keep a connection if there is really ripple from the PD board, even while still?

I agree on the PD board - it is hard to see how it could be at fault.

I think the FMS or some OS or library bug is at fault. But in the interest of thoroughness - the link on the practice field was much much shorter. And I know when we used the practice field we setup in the exact same orientation every time (of course not the case on the field). This leads me to think it worth it to move the robot's radio higher leaving it relatively unobstructed by metal plates or motors.

HTH

[Dusk Star]

The one thing I haven't really seen anyone suggest is malicious intent... I'm not sure how hard it would be (probably not very, as I think the FMS uses WEP) to have an off-field laptop appearing to FMS as the robot. Or just connecting with the same ip as your team, etc. The comments so far have all been from teams with robots that sound like the ones that would have won- is sabotage possible?

Good luck in the future, however! A glitch like that really sucks...

[kjohnson]

Originally Posted by Dusk Star

The one thing I haven't really seen anyone suggest is malicious intent... I'm not sure how hard it would be (probably not very, as I think the FMS uses WEP) to have an off-field laptop appearing to FMS as the robot. Or just connecting with the same ip as your team, etc. The comments so far have all been from teams with robots that sound like the ones that would have won- is sabotage possible?

Good luck in the future, however! A glitch like that really sucks...

Very slim chance considering FMS uses WPA. Thats the purpose of the radio kiosk that all teams must use to re-program their radios at each event. The WPA keys are randomly generated by FMS at each event so you team's key will change between each event. The WPA key list is not available to teams.

[RyanN]

Originally Posted by RufflesRidge

Ryan, any chance you can post the DS logs (they will be chopped on each disconnect) from any of the matches where you has issues? I'm curious how they compare to the ones that Joe posted from 330.

From what I saw on the webcast you guys did move a bit in one of your matches on Saturday morning, that would be a good one to get the logs from if possible.

Lucky you, I just wrote an epic email with screenshots of our logs.

I just got through looking at the log files from my MacBook Pro, the Dell, and the Classmate. Everything that I see is pointing to network interference, not related to us. I'm int talks with a person from NI that designed the driver station software and created the logger. I'm supposed to talk with him on the phone sometime today.

I also looked through the log from the 20 minutes we drove on the practice field. We had some latency, some packet loss, and we did have 2 very small, short glitches, which I noticed, and they did show up in the log, but the important thing to note is that we regained control immediately, and actually never lost communication, it was some weird glitch where we would go to disabled for just a split second, the shooter would shut off (which is what I noticed), and then it came right back to teleoperated control.

Looking at field data from Thursday when we practiced, I saw no issues with latency, packet loss, or anything. We actually ran fine Thursday, it was mechanical problems that disabled us that match.



Friday, every single match we played had significantly higher latency and packet loss. What was different? More people with smart phones? All their systems were running?

Here's a sample from one match on Friday taken with my laptop:



Notice before the match starts, the CPU usage is really high as the system boots and gets everything ready to run, that seems normal to me. Once the match actually starts though, CPU usage is pretty normal, and actually really good for an FRC robot. Then we start dropping packets and lose battery voltage for a few seconds. We only run the match for little over a minute until we lose all communication. No battery spikes, no high CPU usage, nothing. This indicates to me the field dropped us after seeing the missed packets. The steady line for the latency (green) when we start dropping packets indicates they could no longer communicate with our robot, as the latency never stays the same. Another indication is the missing battery voltage. Then after 30 seconds, just as the game was ending, we regained control for a split second, then the match ended. Something smells fishy here... this happened multiple times, not just this once.

At the end of Friday, we ran a match with us on the stand. No one else was there, just Fusion and some of the FTA. Crowd was gone, the rest of the people at the FTA Table were gone. Here is what that match looked like:



The code running at this time was our stripped code, but with CAN implemented still. Notice before they ran us, we were connected for a long time while Steve and the FTA guy discussed the situation. Never once did we lose a connection, and we had very low packet loss. We started the match in autonomous, CPU came up as expected, and we transitioned to teleop without any problems and ran until our chain came off, when the voltage became stable. Not once did we lose a connection after the match started. Now let's go to the next morning, first match of the day. Nothing on the robot had changed.



Continued below...

[RyanN]

Continued from above...

We drop packets, lose communication multiple times. Shouldn't this be enough to prove our point that it was not us? It ran fine with no one else there, but didn't run at all when people were there? And again, we were never offered the alternative router to use, and when Brandon Higgs asked, we were denied use of it.

Here's the second match of the day. Same code, but now we're using the classmate.



Nothing had changed. When we missed those balls during autonomous, it was because we lost communication with the robot... We maintained teleoperated for approximately 15 seconds before we dropped. Later in the match, we regained control for a few seconds. Then the match ended. After the match ended, we lost communication for a few seconds, then regained communication.

Our third match Saturday was bypassed due to an electrical problem we created between match 2 and 3 while replacing our 12V-5V converter. The router was never powered on, and we were bypassed that match.

Here was our fourth and final match of the day. CAN has been taken out at this point. We're running basically bare stock code. No camera, no CAN, no closed loop control. We had kit-bot programming in our Mercedes of a robot.



We still dropped communication like crazy and didn't maintain control during the match.

Finally, we ran on the practice field in front of the FTA guy that was less than helpful to us for 20 minutes, scored 30/32 balls, and had two small glitches, but they lasted no longer than a fraction of a second and we regained control. We places our 'Mercedes' code back on it, and used the Dell again.

We're still reviewing data and trying to piece things together.

[jspatz1]

With all the reported trouble with this issue, including our own, there seems to be a problem out there. I have not seen much discussion of results with the rev. B router in competition. Can anyone report the same trouble with a rev. B router? We will be running a rev.B 1522 at St. Louis this weekend (wk 4), so we will report on what we learn.

[RyanN]

Originally Posted by nukemknight

Very slim chance considering FMS uses WEP. Thats the purpose of the radio kiosk that all teams must use to re-program their radios at each event. The WEP keys are randomly generated by FMS at each event so you team's key will change between each event. The WEP key list is not available to teams.

We've considered it. I'm curious as to the WEP key list. You said it's generated at each event, so it's not recreated each time you reprogram the router.

I talked with the student that reprogrammed our router. Apparently all you have to do is enter in your team number, and that's it?

So another team could simply bring their spare router, tell the kiosk that they're #364, and they would have our encryption key? Messing us up would be as easy as leaving the router powered on in the pit under some jackets or something, right? That would explain why it would have worked after everyone left on Friday. Power to the pits I think was already cut off. It wouldn't jam our signal on the practice field because we're using their network equipment.

I don't want to believe it is malicious, but it's hard to brush off. FTA was completely unwilling to help us out in regards of getting the spare router, and we even asked for a different (temporary) team number, and they refused to do that for us.

[Kevin Sevcik]

Ryan,

First, it's a WPA key. Second, the kiosk only sets up the radio for bridge mode. I don't have one here to verify, but I'm pretty sure switching over to AP mode wouldn't end up with a fully configured AP that your bridge would try to connect to. I could be wrong on that point, though. An identical radio in bridge mode shouldn't cause problems unless it's hooked up to something with a conflicting IP or something. At any rate, I'm really doubtful on that one. It just seems so unlikely that someone would go to that trouble. Incompetence seems unlikely as well, as who would accidentally program a radio with the wrong number and then leave it plugged in? Still, the self-serve WPA kiosk is definitely a gaping security hole in the system. I wouldn't advocate going back to the bad old days of handing out and programming WPA keys by hand, but handing out a team specific password for the kiosk would be a good idea.

Lastly, when you come to Lone Star, would you consider leaving everything as is for a practice match? I'm pretty sure Lone Star isn't getting the Bayou truck, so it'd be a pretty compelling data point if your robot, as is, does/doesn't work on the Lone Star field.

[RyanN]

Originally Posted by Kevin Sevcik

Ryan,

First, it's a WPA key. Second, the kiosk only sets up the radio for bridge mode. I don't have one here to verify, but I'm pretty sure switching over to AP mode wouldn't end up with a fully configured AP that your bridge would try to connect to. I could be wrong on that point, though. An identical radio in bridge mode shouldn't cause problems unless it's hooked up to something with a conflicting IP or something. At any rate, I'm really doubtful on that one. It just seems so unlikely that someone would go to that trouble. Incompetence seems unlikely as well, as who would accidentally program a radio with the wrong number and then leave it plugged in? Still, the self-serve WPA kiosk is definitely a gaping security hole in the system. I wouldn't advocate going back to the bad old days of handing out and programming WPA keys by hand, but handing out a team specific password for the kiosk would be a good idea.

Lastly, when you come to Lone Star, would you consider leaving everything as is for a practice match? I'm pretty sure Lone Star isn't getting the Bayou truck, so it'd be a pretty compelling data point if your robot, as is, does/doesn't work on the Lone Star field.

I'm doubtful on it as well. We're planning on leaving everything the same until we know what we're doing. We're not going to unbag it until we know if we're not going to Lone Star.

I just spoke with Greg on the phone for a nearly an hour. I sent him the email I sent above, and talked with him about all what was going on.

He doesn't have much to say yet. He is not sure what is causing the packet loss right before autonomous, or the short chunks of data that are going missing, as it's not enough time to mean a power problem. He said a bad ethernet problem could cause it. I don't believe that to be our case as we did pretty extensive testing to be sure that we were electrically good.

[Racer26]

I'd really like to see FRC return to a 900MHz radio solution.

Consumer-electronics grade Wifi is just too widespread to count on there being little-to-no-interference.

We're all carrying in literally hundreds of 802.11 wifi-enabled devices that talk in the 2.4GHz and 5GHz bands, never mind that most of the venues have at least one of their own.

[Captaindan]

Can cell phones really use the bandwith on the field? (Via the the request for network usage) Last year we were told to shut off wireless devices. Now more then ever with cellphone usage up tenfold from last year is this the problem? Were we the the unlucky team with a 2.4 or 5 mhz bandwith?