Team Fusion #364, Bayou Regional, FMS Woes

[otherguy]

2168 seemed to have this problem at NYC.

We never had an issue, period. Not for the entire competition until finals. We went 8-0-0, ranked #1, and had the highest teleop points. Our robot works, and we showed that. That's what makes this so unbelievably frustrating.

During our first semi-finals match... all three robots lost communication on the field. 2168 and 329 came back up but 1676 was disabled all match. The blue team did not see such an event. One of our mentors spent 10 minutes with the FMS manager diagnosing what was causing the problem. He said: "There was nothing they would do but it was very evident from the data the FMS collects that all robots were having 200 - 700ms delays."

1676 changed their bridge and Ethernet cable, and 329 deployed new code.
2168 verified wiring and swapped out one of our Jags which seemed to be failed on the drive train*** and the battery that we used during that match was getting way too low ~7V during the match. It was marked as good. (It is now marked as failed). We checked frantically for shorts but our chassis (as it had been all competition) showed infinite resistance to ground and pwr. The battery cable to the andersen connector was noticeably warm to the touch, andersen to the PD board was not however...

***Later we found, during pit testing, that the snap action breaker at that position seemed to be failing open. We have not yet tested the Jag we thought to be failed, it may have been fine, just not receiving power.

ultimately during our second semi-final match we had intermittent comms again. The lift motors were non-operational and since there was a loss of comm issues the drivetrain was not responsive.

Something to note, our team uses victors on the lift (pwm to digital sidecar on slot 2 of 4 slot crio). All other motors are CAN Jaguars connected to 4slot crio via DB9 connection. All 6 Jaguars remained operation during the matches. The victors were apparently not receving commands, however I was on the field right next to our robot and could see the light on the victors were solid, so PWM signals were apparently being sent?

All testing between matches on the side of the field showed all systems being fully functional. No faults with CAN traffic.

After jerking around in semi-2 we brought the robot back to the pit to try to figure out what could have been going wrong. Our operator noted that our button box (psoc3 FIRSTouch IO Module) was acting "weird". Typically in the IO tab on the driver station we should see all 16 Digital IOs pulled high. All indicators were extinguished, and not changing state with button presses, every now and then random ones would flicker on and off. Analog channels appeared to be responding properly to potentiometer changes.
We're using the classmate for the driverstation. It had not been restarted since the semi match ended.

Wiring for the board is all potted in glue, so its unlikely that it would have failed randomly. We checked wiring however and wiggling the usb connection at the psoc seemed to cause a fluttering of indications on the driver station. So we thought we had a wiring problem. I didn't believe we had a random failure of that board (The usb cable is knotted before leaving the box to prevent any damage inside to cabling), so I plugged it into my personal laptop and opened the driver station. Everything worked flawlessly. I plugged it back into the classmate and the same issue presented itself as before.
When the power cable was plugged into the classmate the indicators fluttering seemed to be more erradic which led me to think there was a noise issue on the classmate usb port or the hub we are using.
Restarting classmate (including battery removal) resolved the issue however, and it couldn't be replicated again.

We were able to replicate the field issue (the victors not responding - no ball lift) once in our pit, but over an hour of testing after that revealed everything to be operational.

We've come up with a list of items to check/replace at our next regional. PD board is at the top of my list, but that doesn't explain the weirdness we saw with the PSOC.

Has anyone had problems with the PSOC this year? Has anyone had success with other USB IO Modules? I know there was one which emulated a joystick (can't remember the name/seller) posted earlier in the season from a smaller robotics site.

Hopefully we can shed some light on what's going on here and come to a resolution. Good luck to everyone in fixing the issues on your bots. Keep us updated.

[Greg McKaskle]

Team 624, I may know what your issue is. I'm still looking into a detailed fix, but I'd like someone from the team, ideally a mentor, to PM me. Sorry if this sounds secretive, but I saw something at WPI, and I just tracked it down Saturday morning. I want to get a few more data points at this point. And no, this isn't causing all field comms issues.

Greg McKaskle

[MikeZ]

Greg

I will PM you later this morning. Thanks

[thefro526]

Originally Posted by otherguy

Has anyone had problems with the PSOC this year? Has anyone had success with other USB IO Modules? I know there was one which emulated a joystick (can't remember the name/seller) posted earlier in the season from a smaller robotics site.

In 2010, 816 had numerous issues with the PSOC board - we'd randomly die out mid match, lose one side of our drive the next, and then lose the entire drive but retain all of our aux functions the next. Our 'easy solution' was to unplug the PSOC and replace the custom part of the DS with a game pad. After the season we replaced the PSOC with a CCI from eStop and never had any issues with that robot or our 2011 machine: http://estoprobotics.com/estore/inde...d&productId=33

[Gary Dillard]

SPAM had dropped coms issues almost every match at the Orlando regional, they changed out 2 PD boards which were indicating some kind of loss of ground and it still didn't fix the problem. I don't know all the details - Eric was texting me during the competition - but it's starting to smell awfully fishy.

[RyanN]

Originally Posted by Gary Dillard

SPAM had dropped coms issues almost every match at the Orlando regional, they changed out 2 PD boards which were indicating some kind of loss of ground and it still didn't fix the problem. I don't know all the details - Eric was texting me during the competition - but it's starting to smell awfully fishy.

Here's my stance on it the PD board. Besides it making no sense... how is it that a consumer grade Linksys router (the one provided on the practice field) is able to keep a connection, while FIRST's field system using high quality equipment cannot keep a connection if there is really ripple from the PD board, even while still?

If we do get to Lone Star, we will be swapping out the PD board, because that is the last thing to swap. I'm also loading our stripped down code and going bare minimal for at least our first match for Thursday to verify we're all good to go, then increase the complexity throughout the day.

[RufflesRidge]

Ryan, any chance you can post the DS logs (they will be chopped on each disconnect) from any of the matches where you has issues? I'm curious how they compare to the ones that Joe posted from 330.

From what I saw on the webcast you guys did move a bit in one of your matches on Saturday morning, that would be a good one to get the logs from if possible.

[Jared Russell]

When you guys run wirelessly on the practice field, are you using WPA/WEP? That is the only potential difference between practice and FMS on the robot side. Perhaps the regulated 12V supply to the wireless bridge is affecting encrypted operations...?

I know the radio configuration software has changed since last season, and I suspect on average robots are far more bandwidth hungry than in the past (lots of cameras streaming back to Dashboards). Those might be two other factors causing the failure when other robots are active?

What a heartbreaking thread to read.

[RyanN]

Originally Posted by RufflesRidge

Ryan, any chance you can post the DS logs (they will be chopped on each disconnect) from any of the matches where you has issues? I'm curious how they compare to the ones that Joe posted from 330.

From what I saw on the webcast you guys did move a bit in one of your matches on Saturday morning, that would be a good one to get the logs from if possible.

Lucky you, I just wrote an epic email with screenshots of our logs.

I just got through looking at the log files from my MacBook Pro, the Dell, and the Classmate. Everything that I see is pointing to network interference, not related to us. I'm int talks with a person from NI that designed the driver station software and created the logger. I'm supposed to talk with him on the phone sometime today.

I also looked through the log from the 20 minutes we drove on the practice field. We had some latency, some packet loss, and we did have 2 very small, short glitches, which I noticed, and they did show up in the log, but the important thing to note is that we regained control immediately, and actually never lost communication, it was some weird glitch where we would go to disabled for just a split second, the shooter would shut off (which is what I noticed), and then it came right back to teleoperated control.

Looking at field data from Thursday when we practiced, I saw no issues with latency, packet loss, or anything. We actually ran fine Thursday, it was mechanical problems that disabled us that match.



Friday, every single match we played had significantly higher latency and packet loss. What was different? More people with smart phones? All their systems were running?

Here's a sample from one match on Friday taken with my laptop:



Notice before the match starts, the CPU usage is really high as the system boots and gets everything ready to run, that seems normal to me. Once the match actually starts though, CPU usage is pretty normal, and actually really good for an FRC robot. Then we start dropping packets and lose battery voltage for a few seconds. We only run the match for little over a minute until we lose all communication. No battery spikes, no high CPU usage, nothing. This indicates to me the field dropped us after seeing the missed packets. The steady line for the latency (green) when we start dropping packets indicates they could no longer communicate with our robot, as the latency never stays the same. Another indication is the missing battery voltage. Then after 30 seconds, just as the game was ending, we regained control for a split second, then the match ended. Something smells fishy here... this happened multiple times, not just this once.

At the end of Friday, we ran a match with us on the stand. No one else was there, just Fusion and some of the FTA. Crowd was gone, the rest of the people at the FTA Table were gone. Here is what that match looked like:



The code running at this time was our stripped code, but with CAN implemented still. Notice before they ran us, we were connected for a long time while Steve and the FTA guy discussed the situation. Never once did we lose a connection, and we had very low packet loss. We started the match in autonomous, CPU came up as expected, and we transitioned to teleop without any problems and ran until our chain came off, when the voltage became stable. Not once did we lose a connection after the match started. Now let's go to the next morning, first match of the day. Nothing on the robot had changed.



Continued below...

[RyanN]

Continued from above...

We drop packets, lose communication multiple times. Shouldn't this be enough to prove our point that it was not us? It ran fine with no one else there, but didn't run at all when people were there? And again, we were never offered the alternative router to use, and when Brandon Higgs asked, we were denied use of it.

Here's the second match of the day. Same code, but now we're using the classmate.



Nothing had changed. When we missed those balls during autonomous, it was because we lost communication with the robot... We maintained teleoperated for approximately 15 seconds before we dropped. Later in the match, we regained control for a few seconds. Then the match ended. After the match ended, we lost communication for a few seconds, then regained communication.

Our third match Saturday was bypassed due to an electrical problem we created between match 2 and 3 while replacing our 12V-5V converter. The router was never powered on, and we were bypassed that match.

Here was our fourth and final match of the day. CAN has been taken out at this point. We're running basically bare stock code. No camera, no CAN, no closed loop control. We had kit-bot programming in our Mercedes of a robot.



We still dropped communication like crazy and didn't maintain control during the match.

Finally, we ran on the practice field in front of the FTA guy that was less than helpful to us for 20 minutes, scored 30/32 balls, and had two small glitches, but they lasted no longer than a fraction of a second and we regained control. We places our 'Mercedes' code back on it, and used the Dell again.

We're still reviewing data and trying to piece things together.

[slijin]

Originally Posted by RyanN

We're still reviewing data and trying to piece things together.

Did you see this post in 330's thread?

Originally Posted by MrClintBarnes

I was FTAA at the regional at Utah this last week, and we were having similar problems. The FTA found that every team on the field had high trip times (>30ms) when there was a team with a D-Link running firmware version 1.4.

We has the CSA downgrade those bridges to v1.2 of the firmware and all of the trip times went back down to normal. I am not sure if this is the same problem that everyone else was having, but downgrading the firmware on any of the older bridges running 1.4 seemed to help.

-Clint

Just to throw that out there.

[abbyjcox]

Whatever the problem is....I just hope that it is fixed IF/when we go to Lonestar...I know that EVERYONE on the team worked our butts off to build what we all agree is the best robot we have ever made.. and it stinks beyond belief to see it all go to waste over a problem that wasn't even our fault.. Personally I am kinda disappointed with FIRST for how unhelpful they were, they above all others should be the prime example of gracious professionalism and this time they fell short.. However, I am very proud of my team mates and how they conducted themselves through all of our issues and I know that even if we are not able to compete and put all of our hard work to good use, that all that hard work was not completely wasted because we all learned A LOT this year, and still had fun in the process despite the hard time we encountered at the regional.

[wireties]

Originally Posted by RyanN

Here's my stance on it the PD board. Besides it making no sense... how is it that a consumer grade Linksys router (the one provided on the practice field) is able to keep a connection, while FIRST's field system using high quality equipment cannot keep a connection if there is really ripple from the PD board, even while still?

I agree on the PD board - it is hard to see how it could be at fault.

I think the FMS or some OS or library bug is at fault. But in the interest of thoroughness - the link on the practice field was much much shorter. And I know when we used the practice field we setup in the exact same orientation every time (of course not the case on the field). This leads me to think it worth it to move the robot's radio higher leaving it relatively unobstructed by metal plates or motors.

HTH

[Dusk Star]

The one thing I haven't really seen anyone suggest is malicious intent... I'm not sure how hard it would be (probably not very, as I think the FMS uses WEP) to have an off-field laptop appearing to FMS as the robot. Or just connecting with the same ip as your team, etc. The comments so far have all been from teams with robots that sound like the ones that would have won- is sabotage possible?

Good luck in the future, however! A glitch like that really sucks...

[kjohnson]

Originally Posted by Dusk Star

The one thing I haven't really seen anyone suggest is malicious intent... I'm not sure how hard it would be (probably not very, as I think the FMS uses WEP) to have an off-field laptop appearing to FMS as the robot. Or just connecting with the same ip as your team, etc. The comments so far have all been from teams with robots that sound like the ones that would have won- is sabotage possible?

Good luck in the future, however! A glitch like that really sucks...

Very slim chance considering FMS uses WPA. Thats the purpose of the radio kiosk that all teams must use to re-program their radios at each event. The WPA keys are randomly generated by FMS at each event so you team's key will change between each event. The WPA key list is not available to teams.