View Full Version : Team Fusion #364, Bayou Regional, FMS Woes
I'm not sure how many people have had issues at the Bayou Regional, but we haven't been able to have a single good match at the Bayou where we haven't disconnected from FMS. They verified that our router remains connected, but they are saying our cRIO keeps dropping off.
We have replaced everything, and I mean everything regarding the control system. Electrically we're good. Our best guess is that the cRIO is crashing mid match.
Here's the thing we cannot get past. Our code works perfectly tethered and on the practice field using wireless. We can beat the heck out of our robot, and it remains connected. CPU usage hovers around 40%, memory around 40%, we have tons of flash space free. When we put our robot on the field, we're always the last to connect to the FMS, we've had it drop and reconnect multiple times while the field is being setup. The field management bypassed us one match and told us we couldn't play because they couldn't keep a connection.
My question is why? I know we weren't the only ones with the issue today. Kryptonite had the same problem during the same match we were in one time. Another team came to talk to FTA after the award ceremony about the same issue. FTA still claims that they don't see any issues with their system. Maybe they don't see any, but if our code runs perfectly everywhere but the field, that makes me believe there is a problem on the side of the code we cannot touch (cRIO Image, and FMS).
We can sometimes make it through autonomous, and during the match, our robot will come in and out of communication. We finally remained connected for at least half match in our last match of the day, and we did great.
I hate to see our kids' work that they spent 6 hard weeks working on go to waste because of a dumb bug or glitch with the cRIO image.
I know there was a team update on 'lightening' code, but we're not running anything more complicated that what we ran in 2008 on a PIC18 microcontroller, which is way less powerful than the cRIO we're given today. We tried lightening the code during the day between matches, and we finally finished right at 7PM today. We have one match left, and it's the only match we can try to redeem ourselves. We have no closed loop control, and I'm calling it our dumb code, because all it does is take joystick output and apply it to motors. Nothing neat, no safety interlocks, no special features. All of the hard work we worked on is useless now because of a problem that we cannot diagnose outside of the FMS system.
Honestly, I'm exhausted. We've spent 2 days now working on this problem. Our CTA, MJ, looked through all of our code, witnessed it working in the pit, and agreed with us that our code works. We have sat in every match so far. We have a very capable robot, and we know it. It's so hard to accept that we basically missed every match this year for a 12 year old team.
Thanks,
Ryan N
I would also like to add that I have over a year of professional LabVIEW experience. Our code can be found here: http://www.chiefdelphi.com/forums/showthread.php?t=103902
I know this is probably obvious, but is there any chance your radio is in Auto or AP mode?
Is your radio mounted near motors, and anything else that could cause interference?
Is it buried deep within the robot, possibly shielded from the wireless signal?
Was the router used a lot last year? A bit beaten up? Could any foreign material have fallen into it? Could it just be dirty inside?
Is the power connector to the radio nice and tight? Hot glue, rubber bands or velcro straps could help here.
These are all a bit of a reach, but at this point, I'm sure you're willing to try anything!
Dear Ryan
First off, I'm really sorry to hear that you and your team had so many difficulties this week at the regional. I sincerely hope that you can have a strong showing in your last match and get the chance to compete in the eliminations.
I would love to hear more about the technical problems you experienced, and hopefully offer a solution. I know you may have already tried what I will suggest below, but I hope you can check this message before tomorrow and that something in my post may solve your problems.
We had this issue last year at chesapeake regional, and it was ultimately fixed by turning off the wireless on our classmate pc. We had intermittent connectivity, much like what you are describing, and then when the wireless was turned off, the robot worked fine. Our symptoms were much like yours, bad connection, and losing coms for almost the entire match. That may be a quick fix if you haven't tried it, although I'm sure you guys have gone through the motions and thoroughly checked everything, and it seems like your router is staying connected, but I wanted to throw this out there. It took us a whole day of trying to work this out last year! ;)
Today, at NYC regional, we had a similar problem which we now have solved. To begin with, I'll describe the problem. We would have coms (albeit after much longer than it normally takes to connect to the field) and then promptly lose communication as soon as we began to drive our robot in tele-op. We were not doing anything fancy that would drop our voltage low enough to reboot the crio ( which would result in about a 45s delay) and we would get erratic times for the loss of communication ranging from 15 seconds, to longer than a minute. This suggested that the problem wasn't something to do with the crio rebooting. We did notice that we were dropping packets.
The FTA, after we spent the day trying different things, and swapping out components, suggested we replace all the wiring to/from the wireless bridge, in our case, specifically an ethernet cable from the bridge to the 2Can. I believe doing this solved our problem, I guess we'll see for sure tomorrow if it worked, but we were finally able to connect successfully. Like you guys, both years we were perfectly fine running tethered.
Let me know if this works for you tomorrow. I can type up a more detailed report, including specifics on what we experienced if you send me a PM.
Once again, I'm really sorry to hear that the communications issues caused you guys to not be able to show what your robot can really do. Best of luck!
Is the router clear of motors and things that might interfere with it? We had to move ours because FMS didn't like the fact we put it low, near the drive motors.
Try a new cRIO? Image it?
Not really sure what it could be just throwing out some ideas. Issues like this always suck, I remember FMS costing us a match or two last year.
Good luck to your team, and I hope you guys get it sorted.
I know this is probably obvious, but is there any chance your radio is in Auto or AP mode?
Is your radio mounted near motors, and anything else that could cause interference?
Is it buried deep within the robot, possibly shielded from the wireless signal?
Our radio is connected below our shooter motors, but we're losing connection while disabled as well, so this is not caused by EMF.
Dear Ryan
First off, I'm really sorry to hear that you and your team had so many difficulties this week at the regional. I sincerely hope that you can have a strong showing in your last match and get the chance to compete in the eliminations.
I would love to hear more about the technical problems you experienced, and hopefully offer a solution. I know you may have already tried what I will suggest below, but I hope you can check this message before tomorrow and that something in my post may solve your problems.
We had this issue last year at chesapeake regional, and it was ultimately fixed by turning off the wireless on our classmate pc. (Wireless has been off.)We had intermittent connectivity, much like what you are describing, and then when the wireless was turned off, the robot worked fine. Our symptoms were much like yours, bad connection, and losing coms for almost the entire match. That may be a quick fix if you haven't tried it, although I'm sure you guys have gone through the motions and thoroughly checked everything, and it seems like your router is staying connected, but I wanted to throw this out there. It took us a whole day of trying to work this out last year! ;)
Today, at NYC regional, we had a similar problem which we now have solved. To begin with, I'll describe the problem. We would have coms (albeit after much longer than it normally takes to connect to the field) and then promptly lose communication as soon as we began to drive our robot in tele-op. We were not doing anything fancy that would drop our voltage low enough to reboot the crio ( which would result in about a 45s delay) and we would get erratic times for the loss of communication ranging from 15 seconds, to longer than a minute. This suggested that the problem wasn't something to do with the crio rebooting. We did notice that we were dropping packets.
The FTA, after we spent the day trying different things, and swapping out components, suggested we replace all the wiring to/from the wireless bridge, in our case, specifically an ethernet cable from the bridge to the 2Can. I believe doing this solved our problem, I guess we'll see for sure tomorrow if it worked, but we were finally able to connect successfully. Like you guys, both years we were perfectly fine running tethered.
Here's what we have replaced:
cRIO
Digital Sidecar
Dlink Router
Dlink Power Cable
Ethernet Cables
I would strongly guess that the problem was a connection, but we're losing communication while disabled and everything. We can run wireless in the practice field and it runs perfectly.
Let me know if this works for you tomorrow. I can type up a more detailed report, including specifics on what we experienced if you send me a PM.
Once again, I'm really sorry to hear that the communications issues caused you guys to not be able to show what your robot can really do. Best of luck!
I will update tomorrow after our match if we run or not. I have a bad feeling that it has been the code the entire time. No reason the code is failing, just that it's too complicated for the FMS to handle for some dumb reason. I know it doesn't make sense, but there's nothing else I can think off as to why it works great everywhere except the field.
GrimmReaper
17-03-2012, 03:30
I'm also at a loss for what could be the cause of your communications issue. I'd be curoious though if you're able to replicate the same issue in the pits while tethered and running FMS light?
you can find a working fms light installer and info here http://208.82.132.46:81/2009PostSeasonReleaseNotes.ashx
direct link to the download: http://208.82.132.46/download/fmslight.setup.msi
i've had good luck with this version on windows 7 but not some of the others... i don't know if it will exhibit the same issues when tethered and with this version of FMS, but it might be a worthwhile test... (quick setup = install, set your ether ip to 10.0.0.5, open FMS, click the big reset button(seems to need it every time you open it fresh), then put your team number on one spot and bypass the others, DS should show FMS control and you should be able to start the match from the FMS panel)
if by chance you ARE able to replicate with FMS tethered in your pit, then immediately try a simple example code if that has issues then it should be something electrical/physical still... if that code is fine but your own code is not - you know where the problem must lie...
hope that helps - good luck and keep us posted.
- Jon
techtiger1
17-03-2012, 05:05
Make sure your not getting something called a watchdog error. There is a way to disable it in the code. I am not by any means a programmer but this has happened to my team before. The symptoms you describe are pretty accurate to what was happening with us. Good luck.
Jack Jones
17-03-2012, 05:58
I've been seeing these symptoms alot, each comes with a smorgasbord of possible causes, and any number of eventual outcomes. There were many when the system was new to us in '09, but there seems to be an increase this year.
One thing I've seen in common is that the electronics were mounted to polycarb sheet. Both '09 and this year the robots are like vandergraph generators (fast moving belts within an isolated shell). Is it possible for a polycarb surface to build up static until the conditions are ripe for it to discharge?
Joe Ross
17-03-2012, 09:00
It sounds like we might be having a similar issue. We've extensively replaced components and code, without a solution (yet). http://www.chiefdelphi.com/forums/showthread.php?threadid=104713
Greg McKaskle
17-03-2012, 09:13
Can I get DS log files from each of you emailed to me. Also can you include a description of the cRIO, the DLink radio particulars including the firmware version. I have your code Ryan. I'm not sure if that is having any impact.
I was at a week one and two regional and haven't seen this. Comms issues have typically been firewall, IP, or electrical issues. I've also seen the issues go away when the radio was reprogrammed or swapped out. I wish I knew what was causing those.
PM me for my gmail account, though you can probably guess what it is.
Greg McKaskle
Joe Ross
17-03-2012, 09:55
We've tried 3 radios, all rev A. Two with firmware version 1.21 (one was ours and one from spare parts) and one with firmware version 1.4 (purchased last month). We've tried 2 cRios, both have been 4 slot cRios (no room for 8 slot).
DonRotolo
17-03-2012, 11:03
At Rutgers our team, along with 75 and 41 all had a similar problem to that described above, except that none of us could connect to the field at all.
Long story short, the automatic radio configuration system was not configuring the radio properly. When we configured it manually (with the explicit approval of the FTA) we were good.
Perhaps checking the radio configuration for oddnesses might help?
Greg McKaskle
17-03-2012, 13:42
Do you know what setting was not good?
Greg McKaskle
And also, can we get a breakdown of what the exact WORKING settings were when you manually re-configured?
These settings will be a good reference for us in the future.
We had issues like these last year and replacing the driver station laptop fixed it.
We, to this day, are still trying to figure out why.
We ran our robot for 20 minutes in the practice field wirelessly. Made 30/30 shots, and it ran flawlessly. Right in front of one of the FTA. I asked him what he thought and he blamed the robot still.
I'm sorry, we worked for 3 days, completely replaced all the control system, replaced our code with basic drive code. It always worked out of the pit, but never on the field.
I call bullcrap on it being us. I'm an experienced LabVIEW programmer, 4th year computer engineering student, and very knowledgeable in electrical. Our robot is wired and configured properly.
FMS cost us every single match. We never ran a single match without disconnection. We would disconnect at any time. Before the match, during autonomous, or during teleop.
I hope FIRST figures out what's going on, because there is nothing more depressing in FIRST to see your robot go on the field knowing it's not going to work.
Tristan Lall
17-03-2012, 16:32
Maybe totally unrelated, but 176 was immobile for long stretches in finals 2 and 3 at Montreal. They worked in autonomous, then stopped dead for the rest of finals 2; then they worked for autonomous in finals 3 and stopped again for about 20 s.
One of their alliance partners was also having some trouble moving around.
(I'm not in Montreal, so I don't know if there were other underlying reasons.)
DonRotolo
17-03-2012, 18:11
Do you know what setting was not good?
Greg McKaskle
When we had the symptom, we could connect to the radio using a browser but there were *no* settings. Noting, like a broken web page. Weird. We hard-reset the radio and could see settings (but they were default, of course). We compared to a good radio and got the WEP code, entered it all manually, and it worked.
We took our second radio, hard reset it, and got it programmed, and it was good, we could see the settings and all. Tried it in our next match, and both radios were OK after that. Go figure.
AllenGregoryIV
17-03-2012, 21:07
I'm also at a loss for what could be the cause of your communications issue. I'd be curoious though if you're able to replicate the same issue in the pits while tethered and running FMS light?
you can find a working fms light installer and info here http://208.82.132.46:81/2009PostSeasonReleaseNotes.ashx
direct link to the download: http://208.82.132.46/download/fmslight.setup.msi
- Jon
The links to FMSlight don't seem to be working does anyone have new ones? I would love to test our control system before our event in two weeks.
Alan Anderson
17-03-2012, 21:12
We ran our robot for 20 minutes in the practice field wirelessly. Made 30/30 shots, and it ran flawlessly. Right in front of one of the FTA. I asked him what he thought and he blamed the robot still.
I'm sorry, we worked for 3 days, completely replaced all the control system, replaced our code with basic drive code. It always worked out of the pit, but never on the field.
I call bullcrap on it being us. I'm an experienced LabVIEW programmer, 4th year computer engineering student, and very knowledgeable in electrical. Our robot is wired and configured properly.
FMS cost us every single match. We never ran a single match without disconnection. We would disconnect at any time. Before the match, during autonomous, or during teleop.
I hope FIRST figures out what's going on, because there is nothing more depressing in FIRST to see your robot go on the field knowing it's not going to work.
The difference between the practice field and the competition field is more than just the presence of FMS. The quality of the wireless signal is going to be very different. Minor RFI issues on your robot, such as noisy power or speed controllers mounted too near the D-Link, have the potential to disrupt match communication without being bad enough to show up during practice.
If you have the D-Link mounted adjacent to anything electical, try separating them. I've seen that fix something that sounds like your symptoms on one team. If you get a chance, try looking at the 5v power going to the D-Link using an oscilloscope. Replacing the Power Distribution Board fixed the same symptoms on another team.
familyguyfreak
17-03-2012, 21:58
Several teams were experiencing communication issues with the field, us included. Ours weren't anything nearly close to team Fusion's but we tried everything, switching Ethernet cables, moving the radio up away from motors. We still had comm issues and brought it up with FTA along with another team and gave them a list of teams that were having the same issues. We never heard back from them but we understood that they were extremely busy. We dropped it because we saw that majority of the teams weren't having problems and just assumed it was us.
Several teams were experiencing communication issues with the field, us included. Ours weren't anything nearly close to team Fusion's but we tried everything, switching Ethernet cables, moving the radio up away from motors. We still had comm issues and brought it up with FTA along with another team and gave them a list of teams that were having the same issues. We never heard back from them but we understood that they were extremely busy. We dropped it because we saw that majority of the teams weren't having problems and just assumed it was us.
I think it has to do with the crowd. Maybe our router is set to a bad channel or something?
FTA let us run our robot Friday night after everyone else left. No one was in the stands, and just a few people were still on the admin table. We ran a full match without any problems at all. We thought we had fixed our problem and packed it up for the night. Came back this morning and we had the first match of the day, and couldn't connect to the field, even while the other 5 robots were off. I think there is too much interference. I'm not sure why we were the only ones affected the entire competition. They just said we had bad network latency and packet loss. We, again, tried 2 different Dlink routers, one was ours, the other was a pit loaner.
So yea, when no one else is near the field, our robot ran fine. With the crowd and everyone working, the robot didn't work.
We were running all PWM, no CAN, no Camera, no closed loop control. Just the basic FRC Robot Framework, with the configuration for our Cypress board, and the basic controls to make our robot work in some fashion. (bare minimum code & it's attached).
What I've ruled out: it's not programming, it's not the cRIO, ethernet, Dlink, power converter, sidecar, NI Modules...
We never replaced our PD board, but replacing it makes no sense. What's the difference with running it wireless at the practice field and running it on the field as far as power is related? Nothing.
Here's our stripped down code. Don't expect anything extravagant at all, it was written & debugged in about an hour, and is meant to be bare minimum.
12341
It's not a programming problem. If it was, it wouldn't work in the pit or the practice field, or even the real field when it does work.
It's also not an electrical problem. We stalled our 4 CIMS, got our voltage down to 7V, and no data loss. We shook, dropped, rattled, beat, and kicked our robot during communication, and again, had no loss. We started up our 4 noisy 550 motors for our shooter to full speed, and started our 4 CIMs at full speed, no load to try to create some bad EMI, and again, no loss or problems.
It's not us. I'm not saying it's the field. I think it's an environmental factor, which isn't our fault, but it is the responsibility of the FTA to control environmental factors. They should be sniffing the air for WiFi hotspots... I know there were tons of WiFi networks being shown. Everyone with a smartphone is polling the air. The ultimate question is why just us? Why were we the only team at the Bayou that constantly had communication failures? Is there something the FTA isn't telling us? Do they have equipment on the 10.3.x.x network for some of their stuff? (Would affect Beachbots 330 too). Conspiracy? I have no idea... I know we weren't the only ones having problems, but I do feel we were cheated out of an equal opportunity during the competition.
Team Fusion all left with a sour taste in our mouth, but we'll manage. FTA did not help us at all, and just stuck with blaming our code (attached), and electrical (explained above). All the mentors are on the same page now. We all know it's not us. We were treated equally. FTA was not able to help anyone really with this issue. When other teams had the issue, they just said to check your code and bandwidth as those are the 'common' problems. I'm upset with FIRST about this. I hate to say this, but this was one of the things that the IFI system didn't have. Someone told me today that IFI can hold huge competitions with dozens of robots running at the same time, without any problems, but FIRST cannot run 6 robots at a time. I'm not blaming any single person or organization, but NI & FIRST needs to get this figured out.
We're not blaming any of us on the team. There were many ideas Thursday and Friday, but after today, we're all confident that our robot is not the issue. It was personally hard for me to remain professional through this whole ordeal. A few of our mentors lost their temper, but I understand why. When we spend 3 days replacing, reconfiguring, and fixing our robot and we yield no results, it gets really frustrating.
Alan Anderson
18-03-2012, 01:13
We never replaced our PD board, but replacing it makes no sense. What's the difference with running it wireless at the practice field and running it on the field as far as power is related? Nothing.
I would normally agree, but replacing the Power Distribution Board did fix the issue for another team having exactly the problem you describe. Yes, it makes no sense. No, there should not be a difference between practice and a competition match.
Consider this: if it's the one thing you haven't changed, and the problem is still there, you can't rule it out.
Let me say this about Team Fusion.
I was heart broken that your bot could not compete at the regional. I saw it at the meetup that you hosted at your school and was ready to watch it on the field.
But - at the same time - I have never been so impressed at a team. Team Fusion, even while not being able to move across the field, showed more sportsmanship than any other team at the event. The whole team cheered for everything from their own team being on the field to another team winning an award. They showed gracious professionalism and good sportsmanship throughout the game.
You made me proud to be in FIRST robotics. To see a team that can be having so many problems still be excited about what they are doing is rare. To see a team that - even reaching the quarterfinals remove themselves from the game for the sake of their alliance members is beyond words.
I hope that my team, Panthrobotics, will emulate yours each year. I think most teams have a lot they can learn from yours.
Kevin Sevcik
18-03-2012, 11:48
Let me say this about Team Fusion.
I was heart broken that your bot could not compete at the regional. I saw it at the meetup that you hosted at your school and was ready to watch it on the field.
But - at the same time - I have never been so impressed at a team. Team Fusion, even while not being able to move across the field, showed more sportsmanship than any other team at the event. The whole team cheered for everything from their own team being on the field to another team winning an award. They showed gracious professionalism and good sportsmanship throughout the game.
You made me proud to be in FIRST robotics. To see a team that can be having so many problems still be excited about what they are doing is rare. To see a team that - even reaching the quarterfinals remove themselves from the game for the sake of their alliance members is beyond words.
I hope that my team, Panthrobotics, will emulate yours each year. I think most teams have a lot they can learn from yours.True story. Anyone who watched the Alliance pairings at Bayou knows that Fusion was picking as the 8th seed, and they immediately called in a replacement so they wouldn't hamstring their alliance. Just before Alliance pairings, Fusion came by my pit, explained what they were going to do and told me that 57 (ranked 18th) was third on their pick list. And then asked me if that was okay, and if I'd be okay with being picked to be on an alliance where the captain would immediately step down. I told them I'd be happy to be picked by them and work on that alliance, and I mean it. We were first pick for #5, but it would've been an honor to play on that #8 alliance for Fusion.
Captaindan
18-03-2012, 12:18
Still there was no solution to our problem. After having com issues i.e. no driving robot almost every match, changing everything from our crio all wires going to each, the radio, sidecar and all wires to and from each. We even changed the power inverter for the radio, checking all the code settings and then running flawlessly in the pit and wirelesly on the practice field in front of the fta. Which i might add we collected balls with the robot and made 30 out of 32 goals in the top, we drove till we were tired of driving, We were again told there was something wrong with our robot rather than a field issue. On top of it all we were ranked 8th as a result of amazing alliances. We ended up picking deserving teams rather than the next best and dropped out as a result of a lack thereof a diagnosis to our problem.
sircedric4
18-03-2012, 14:03
We also had communication problems at the Bayou Regional, with similar symptoms as 364 for our last 2 matches on Friday. We had run all day with no issues, and then all of a sudden we would only work during autonomous modes. We talked to the FTA guy and they told us we had lags greater than 500ms and we were dropping 5000 packets during our matches throughout the day. I believe the average for other teams was less than 20 msec lag and less than 50 dropped packets per match.
We were frustrated that our issues just happened out of nowhere after having an undefeated day. It was at this time that we actually started to hear of 364's similar issues. They seemed to have similar symptoms, and we were one of the teams after the day Friday connecting to the FMS to test our system. Everything worked wireless before we bagged and everything worked tethered. It was only at the regional that we had issues. Overnight I was watching this thread and we did some of the recommendations on here. We got back to working Saturday, but we are not exactly sure which of our numerous changes fixed our issue. We didn't have another bridge or cRio to try, but here is what we did do:
1) Ground tested our system to make sure no electricity was being shorted to the chassis.
2) During ground test, discovered one of our banebot 775's had case shorted sometime after inspection. We replaced it with a new one. (And for future applications we have pretty much decided to go FP motors over the 775)
3) Moved our router/bridge to another location on the robot away from all electrical components and motors.
4) Discovered we had our classmate wifi turned on and turned it off
5) Instituted a complete shutdown of the Classmate after each match to ensure nothing was being held in memory and that our smartdashboard wasn't memory leaking us into oblivion
6) Reduced the resolution on our camera to the lowest setting and dropped the framerate to 15 per second
7) Went back over our C++ code and stripped it down, put alot of our smart stuff like dashboard updates and system polling into a slow do loop that only happened 5 times a second. We put everything but the basic drive functions into that slow loop.
8) Made sure the router is on bridge position before match starts in case it shook to another position
9) Did a lot of praying overnight
We had talked to 364 and they had already done all this with their system so we weren't feeling good coming into Saturday morning about our chances. We however were lucky in that one of those things or some combination fixed our robot enough to be able to finish playing the regional. Our lag dropped down to a bad but acceptable 80 msec average and our packets dropped to 100 average.
We are sorry that 364 had bad issues and could never get their robot to communicate with the field. Please know that the knowledge we gathered from you did help another local team to give the Texas alliance a good run for their money in the finals. Thanks for your gracious professionalism, and I personally hope that your future FIRST outings are more rewarding experiences.
To get back on topic, I want one specific question answered.
What would make us connect and disconnect before the match even began?
Our configuration for the last match was this:
Dumbed down code, no CAN, no PID, just connecting joystick outputs to motors and solenoids.
No camera connected
We had trouble maintaining connection to FMS during the pre-match announcements. Why? No EMI, no vibrations, and we know our power is good from testing all the power and connections in the pit. At this point, all of our hardware had been replaced. Our power test consisted of probing voltages when the robot was still, everything came up clean. We then enabled teleoperated, shook, kicked, dropped, rattled the whole robot, and saw no loss of communication, no dropped packets, and no spikes in latency. I'm confident that the power on our robot is in good shape after when I did to the robot during testing. In no case would our robot be subject to the abuse I gave it.
Even more troubling is why it worked Friday night when no one else was in the venue, and why it didn't work Saturday morning when we were still the only one on the field powered up.
So yea, why would we connect and drop before the match even began? It seriously sounds like a crowded WiFi channel or a resource conflict on the FMS side of things.
Ryan, did you try switching the wireless channel or downloading a utility to see what wireless channels were being used around a field in a typical match? If switching the channel doesn't work then we just need to start building a Faraday cage around the field ;) I hate that you guys had to go through that this year since the robot looked like a mean machine. Are you guys going to be able to attend any other regionals or was that it?
-Brantley
Team Fusion Captain 2007
Questions for someone who knows more about the FMS than I:
Is there a way to tether a robot directly to the competition field FMS, and bypass wireless altogether?
That could've helped rule out whether this is a wireless issue vs code, etc.
If someone/something created another wireless network with the same "364" SSID, could that cause similar types of problems?
I know usually someone is charged with monitoring for rogue wireless networks though.
I wish I had more suggestions for you...
Joe Ross
18-03-2012, 18:40
We had trouble maintaining connection to FMS during the pre-match announcements. Why? No EMI, no vibrations, and we know our power is good from testing all the power and connections in the pit. At this point, all of our hardware had been replaced. Our power test consisted of probing voltages when the robot was still, everything came up clean. We then enabled teleoperated, shook, kicked, dropped, rattled the whole robot, and saw no loss of communication, no dropped packets, and no spikes in latency. I'm confident that the power on our robot is in good shape after when I did to the robot during testing. In no case would our robot be subject to the abuse I gave it.
Even more troubling is why it worked Friday night when no one else was in the venue, and why it didn't work Saturday morning when we were still the only one on the field powered up.
So yea, why would we connect and drop before the match even began? It seriously sounds like a crowded WiFi channel or a resource conflict on the FMS side of things.
Your symptoms sound exactly like ours, except that our problems did not stop when we stayed late on Friday night after everyone else left.
We did monitor the cRIO and radio power with a DMM when the dropouts occurred, and there was no change. However, replacing the power distribution board did fix our problem. I can't explain it, but it worked.
Ryan, did you try switching the wireless channel or downloading a utility to see what wireless channels were being used around a field in a typical match? If switching the channel doesn't work then we just need to start building a Faraday cage around the field ;) I hate that you guys had to go through that this year since the robot looked like a mean machine. Are you guys going to be able to attend any other regionals or was that it?
-Brantley
Team Fusion Captain 2007
We talked about it, but the FMS is so particular about everything. They have it all setup with a kiosk, and the teams aren't supposed to change anything on the router. We couldn't get FTA to budge on much of anything. The fact that it was connecting sometimes was good enough for them to say that the problem was ours and not theirs.
I wish I would have had software to check how busy the channels were. I feel pretty certain our issue was something to do with conflicting resources on the network. That's the only thing that made sense to me why it worked Friday night when no one was there, and didn't work the next morning.
The fact that it worked fine on Friday night, and that we're using such dumbed down code with no features, just basic drive code leads me to believe that the issue is not code related at all. Also, the fact that it runs everywhere but the playing field lets me know that the problem isn't code related.
We're talking with some people about trying to get pushed into the Lone Star Regional real fast.
Despite our hopes and expectations following a full four and half minutes of connection in the morning, Team 846 was unable to maintain our connection to FMS throughout any of our nine matches.
In the morning, we had a few minutes on the field with FMS to test communications, along with another robot. Though things appeared to magically work following changes to task priority in the code, the same (deployed) code did not work in the actual match with six robots on the field.
It is probably useful to note that, according to FMS logs, less than 10% of matches at NYR ran without at least one robot dropping communications.
wireties
18-03-2012, 19:49
We had a few minor latency issues with the Bayou FMS on Friday morning but not after that. Team Fusion had a great robot. It is a travesty they did not get to play! They were definitely robbed.
I agree with the Fusion mentors on this thread, interference of some kind makes the most sense. But I have one (pretty weak) troubleshooting comment - was the radio always mounted in the same location? Did I see a statement about it being mounted "under a motor"? Did you try mounting it high and without any noisy active electronics in close proximity? Were the delays in communications several seconds or 1 minute plus?
Though you replaced the radio many times, did you replace the power and ethernet wiring?
AllenGregoryIV
18-03-2012, 20:44
We're talking with some people about trying to get pushed into the Lone Star Regional real fast.
I really hope this works out, we would love to have you come to Houston. Your robot looked fantastic and it's travesty we never got to see it at its full potential.
A program I have found that works well for viewing wireless channels and possible interference is called inSSIDer (http://www.metageek.net/products/inssider/).
I really hope FIRST can find the cause of these communication issues. The power distribution board and the router configuration seem to be the two things that were never changed. I wonder if configuring your setup as a different team number would have resolved the problem.
I really hope this works out, we would love to have you come to Houston. Your robot looked fantastic and it's travesty we never got to see it at its full potential.
A program I have found that works well for viewing wireless channels and possible interference is called inSSIDer (http://www.metageek.net/products/inssider/).
I really hope FIRST can find the cause of these communication issues. The power distribution board and the router configuration seem to be the two things that were never changed. I wonder if configuring your setup as a different team number would have resolved the problem.
We asked about another, but temporary number and they said they can't do it. The router was moved to the best location available on the robot. It was located under two 550 motors, but has now moved to an unpopulated area velcroed to some lexan above the crio, digital sidecar, and our solenoids.
Jimmy the Kidd
18-03-2012, 21:40
As inexperienced as I am, I still tried to the point of headaches to figure out what the problem could have been for us. I've never been so frustrated in my life, nor more disappointed. However, I felt like the decision we made regarding finals was the best thing we could have done. We'll see what the future holds, what with this being my last year as a student.
Tom Line
18-03-2012, 21:57
I feel for you guys. We've never missed as much of a regional as you did, but we missed the first 1/3 due to control issues in '09, and I know how horrible we felt. I can't imagine being in your situation.
I didn't see it listed anywhere, but I may have missed it: Have you guys gone out with a different driver station and no cypress board to see what happens?
I feel for you guys. We've never missed as much of a regional as you did, but we missed the first 1/3 due to control issues in '09, and I know how horrible we felt. I can't imagine being in your situation.
I didn't see it listed anywhere, but I may have missed it: Have you guys gone out with a different driver station and no cypress board to see what happens?
We went out with 3 different laptops as our driver station.
The old style classmate that we received in 2010.
A Dell M5040 (we personally don't like this one, and it restarted during our first match on Friday due to pending Windows Updates... go figure)
Late 2010 MacBook Pro (running Windows 7 natively)
All three laptops are fully capable of running the Driver Station software along with the Dashboard.
We have not tried running without the cypress board. It's an integral part of our robot design. Running without it would hinder our driver from running. The Cypress board shouldn't be the problem.
On Saturday, when we were desperate, we used the Classmate, with the default dashboard, and the dumbed down code that I've mentioned before. It was a stock FRC system.
Joe, we'll try replacing the PD board when we get a chance. As of right now, we're not registered for another regional. We're going to try to register, and as such, we don't want the students messing with the robot until a final decision has been made.
Elizabeth Waters
19-03-2012, 00:14
Yep, we had a very similar issue several times on the field at Bayou.
We never had any issues connecting to the field before the match began. BUT, after running autonomous, we'd still be "connected" but the driver station would display "No Robot Code." When this happened the first time during a practice match on Thursday, we waited to see if it would find the code; never did. The FTA suggested that next time we try to reboot the cRIO in the middle of the match and see what happened. So we listened: every time we "lost robot code" on the field (which happened at least 4 times over the course of Friday and Saturday), we would immediately reboot the cRIO. What would happen:
1) Auto works fine
2) Teleop begins: "No Robot Code"
3) Driver immediately reboots cRIO = lose connection with FMS temporarily
4) Regain connection within ~5/10 secs.
5) "No Robot Code" displays again
6) Regain comm ("find" code) with about 20 sec. left in match
So ultimately, reboting the cRIO worked for us (if I can really say that... We were still out most of the match, obviously). Did you guys ever try this?
But you're right, this still doesn't answer the question of why this occurred in the first place. Our losses were more sporradic than yours, so not sure why ours worked most of the time. When we talked to the FTA about this issue during quals, he blamed it on our robot, too (and you know, who knows, it very well may be the robot). When it happened again during elims, FTA said it still looked like FMS was fine; they said there were no issues on their end, and when the code was "lost" after auto, everything was still reading fine in relation to their connection with our bot. Who knows.
Hope someone can figure this out soon.... We're also attending Lone Star in a few weeks.
RufflesRidge
19-03-2012, 01:14
Yep, we had a very similar issue several times on the field at Bayou.
Hope someone can figure this out soon.... We're also attending Lone Star in a few weeks.
I don't think this sounds like the same issue based on Ryan's description of 364's problems. It sounds like your code is crashing or sucking up all the CPU.
I would use the Practice match feature on the Driver Station and watch your CPU and memory on the Charts tab of the Driver Station.
I would also look very carefully at all image processing code and make sure you are limiting the number of particles you are processing and catching any exceptions or handling any errors that may be generated including return values of Null from the various vision pieces.
otherguy
19-03-2012, 04:18
2168 seemed to have this problem at NYC.
We never had an issue, period. Not for the entire competition until finals. We went 8-0-0, ranked #1, and had the highest teleop points. Our robot works, and we showed that. That's what makes this so unbelievably frustrating.
During our first semi-finals match... all three robots lost communication on the field. 2168 and 329 came back up but 1676 was disabled all match. The blue team did not see such an event. One of our mentors spent 10 minutes with the FMS manager diagnosing what was causing the problem. He said: "There was nothing they would do but it was very evident from the data the FMS collects that all robots were having 200 - 700ms delays."
1676 changed their bridge and Ethernet cable, and 329 deployed new code.
2168 verified wiring and swapped out one of our Jags which seemed to be failed on the drive train*** and the battery that we used during that match was getting way too low ~7V during the match. It was marked as good. (It is now marked as failed). We checked frantically for shorts but our chassis (as it had been all competition) showed infinite resistance to ground and pwr. The battery cable to the andersen connector was noticeably warm to the touch, andersen to the PD board was not however...
***Later we found, during pit testing, that the snap action breaker at that position seemed to be failing open. We have not yet tested the Jag we thought to be failed, it may have been fine, just not receiving power.
ultimately during our second semi-final match we had intermittent comms again. The lift motors were non-operational and since there was a loss of comm issues the drivetrain was not responsive.
Something to note, our team uses victors on the lift (pwm to digital sidecar on slot 2 of 4 slot crio). All other motors are CAN Jaguars connected to 4slot crio via DB9 connection. All 6 Jaguars remained operation during the matches. The victors were apparently not receving commands, however I was on the field right next to our robot and could see the light on the victors were solid, so PWM signals were apparently being sent?
All testing between matches on the side of the field showed all systems being fully functional. No faults with CAN traffic.
After jerking around in semi-2 we brought the robot back to the pit to try to figure out what could have been going wrong. Our operator noted that our button box (psoc3 FIRSTouch IO Module) was acting "weird". Typically in the IO tab on the driver station we should see all 16 Digital IOs pulled high. All indicators were extinguished, and not changing state with button presses, every now and then random ones would flicker on and off. Analog channels appeared to be responding properly to potentiometer changes.
We're using the classmate for the driverstation. It had not been restarted since the semi match ended.
Wiring for the board is all potted in glue, so its unlikely that it would have failed randomly. We checked wiring however and wiggling the usb connection at the psoc seemed to cause a fluttering of indications on the driver station. So we thought we had a wiring problem. I didn't believe we had a random failure of that board (The usb cable is knotted before leaving the box to prevent any damage inside to cabling), so I plugged it into my personal laptop and opened the driver station. Everything worked flawlessly. I plugged it back into the classmate and the same issue presented itself as before.
When the power cable was plugged into the classmate the indicators fluttering seemed to be more erradic which led me to think there was a noise issue on the classmate usb port or the hub we are using.
Restarting classmate (including battery removal) resolved the issue however, and it couldn't be replicated again.
We were able to replicate the field issue (the victors not responding - no ball lift) once in our pit, but over an hour of testing after that revealed everything to be operational.
We've come up with a list of items to check/replace at our next regional. PD board is at the top of my list, but that doesn't explain the weirdness we saw with the PSOC.
Has anyone had problems with the PSOC this year? Has anyone had success with other USB IO Modules? I know there was one which emulated a joystick (can't remember the name/seller) posted earlier in the season from a smaller robotics site.
Hopefully we can shed some light on what's going on here and come to a resolution. Good luck to everyone in fixing the issues on your bots. Keep us updated.
Greg McKaskle
19-03-2012, 07:24
Team 624, I may know what your issue is. I'm still looking into a detailed fix, but I'd like someone from the team, ideally a mentor, to PM me. Sorry if this sounds secretive, but I saw something at WPI, and I just tracked it down Saturday morning. I want to get a few more data points at this point. And no, this isn't causing all field comms issues.
Greg McKaskle
Greg
I will PM you later this morning. Thanks
thefro526
19-03-2012, 07:54
Has anyone had problems with the PSOC this year? Has anyone had success with other USB IO Modules? I know there was one which emulated a joystick (can't remember the name/seller) posted earlier in the season from a smaller robotics site.
In 2010, 816 had numerous issues with the PSOC board - we'd randomly die out mid match, lose one side of our drive the next, and then lose the entire drive but retain all of our aux functions the next. Our 'easy solution' was to unplug the PSOC and replace the custom part of the DS with a game pad. After the season we replaced the PSOC with a CCI from eStop and never had any issues with that robot or our 2011 machine: http://estoprobotics.com/estore/index.php?_a=viewProd&productId=33
Gary Dillard
19-03-2012, 08:52
SPAM had dropped coms issues almost every match at the Orlando regional, they changed out 2 PD boards which were indicating some kind of loss of ground and it still didn't fix the problem. I don't know all the details - Eric was texting me during the competition - but it's starting to smell awfully fishy.
SPAM had dropped coms issues almost every match at the Orlando regional, they changed out 2 PD boards which were indicating some kind of loss of ground and it still didn't fix the problem. I don't know all the details - Eric was texting me during the competition - but it's starting to smell awfully fishy.
Here's my stance on it the PD board. Besides it making no sense... how is it that a consumer grade Linksys router (the one provided on the practice field) is able to keep a connection, while FIRST's field system using high quality equipment cannot keep a connection if there is really ripple from the PD board, even while still?
If we do get to Lone Star, we will be swapping out the PD board, because that is the last thing to swap. I'm also loading our stripped down code and going bare minimal for at least our first match for Thursday to verify we're all good to go, then increase the complexity throughout the day.
RufflesRidge
19-03-2012, 10:39
Ryan, any chance you can post the DS logs (they will be chopped on each disconnect) from any of the matches where you has issues? I'm curious how they compare to the ones that Joe posted from 330.
From what I saw on the webcast you guys did move a bit in one of your matches on Saturday morning, that would be a good one to get the logs from if possible.
Jared Russell
19-03-2012, 11:17
When you guys run wirelessly on the practice field, are you using WPA/WEP? That is the only potential difference between practice and FMS on the robot side. Perhaps the regulated 12V supply to the wireless bridge is affecting encrypted operations...?
I know the radio configuration software has changed since last season, and I suspect on average robots are far more bandwidth hungry than in the past (lots of cameras streaming back to Dashboards). Those might be two other factors causing the failure when other robots are active?
What a heartbreaking thread to read.
wireties
19-03-2012, 11:32
Here's my stance on it the PD board. Besides it making no sense... how is it that a consumer grade Linksys router (the one provided on the practice field) is able to keep a connection, while FIRST's field system using high quality equipment cannot keep a connection if there is really ripple from the PD board, even while still?
I agree on the PD board - it is hard to see how it could be at fault.
I think the FMS or some OS or library bug is at fault. But in the interest of thoroughness - the link on the practice field was much much shorter. And I know when we used the practice field we setup in the exact same orientation every time (of course not the case on the field). This leads me to think it worth it to move the robot's radio higher leaving it relatively unobstructed by metal plates or motors.
HTH
Dusk Star
19-03-2012, 13:11
The one thing I haven't really seen anyone suggest is malicious intent... I'm not sure how hard it would be (probably not very, as I think the FMS uses WEP) to have an off-field laptop appearing to FMS as the robot. Or just connecting with the same ip as your team, etc. The comments so far have all been from teams with robots that sound like the ones that would have won- is sabotage possible?
Good luck in the future, however! A glitch like that really sucks...
kjohnson
19-03-2012, 13:24
The one thing I haven't really seen anyone suggest is malicious intent... I'm not sure how hard it would be (probably not very, as I think the FMS uses WEP) to have an off-field laptop appearing to FMS as the robot. Or just connecting with the same ip as your team, etc. The comments so far have all been from teams with robots that sound like the ones that would have won- is sabotage possible?
Good luck in the future, however! A glitch like that really sucks...
Very slim chance considering FMS uses WPA. Thats the purpose of the radio kiosk that all teams must use to re-program their radios at each event. The WPA keys are randomly generated by FMS at each event so you team's key will change between each event. The WPA key list is not available to teams.
Ryan, any chance you can post the DS logs (they will be chopped on each disconnect) from any of the matches where you has issues? I'm curious how they compare to the ones that Joe posted from 330.
From what I saw on the webcast you guys did move a bit in one of your matches on Saturday morning, that would be a good one to get the logs from if possible.
Lucky you, I just wrote an epic email with screenshots of our logs.
I just got through looking at the log files from my MacBook Pro, the Dell, and the Classmate. Everything that I see is pointing to network interference, not related to us. I'm int talks with a person from NI that designed the driver station software and created the logger. I'm supposed to talk with him on the phone sometime today.
I also looked through the log from the 20 minutes we drove on the practice field. We had some latency, some packet loss, and we did have 2 very small, short glitches, which I noticed, and they did show up in the log, but the important thing to note is that we regained control immediately, and actually never lost communication, it was some weird glitch where we would go to disabled for just a split second, the shooter would shut off (which is what I noticed), and then it came right back to teleoperated control.
Looking at field data from Thursday when we practiced, I saw no issues with latency, packet loss, or anything. We actually ran fine Thursday, it was mechanical problems that disabled us that match.
http://i.imgur.com/a6TL0l.png (http://imgur.com/a6TL0)
Friday, every single match we played had significantly higher latency and packet loss. What was different? More people with smart phones? All their systems were running?
Here's a sample from one match on Friday taken with my laptop:
http://i.imgur.com/k7n01l.png (http://imgur.com/k7n01)http://i.imgur.com/2O8twl.png (http://imgur.com/2O8tw)
Notice before the match starts, the CPU usage is really high as the system boots and gets everything ready to run, that seems normal to me. Once the match actually starts though, CPU usage is pretty normal, and actually really good for an FRC robot. Then we start dropping packets and lose battery voltage for a few seconds. We only run the match for little over a minute until we lose all communication. No battery spikes, no high CPU usage, nothing. This indicates to me the field dropped us after seeing the missed packets. The steady line for the latency (green) when we start dropping packets indicates they could no longer communicate with our robot, as the latency never stays the same. Another indication is the missing battery voltage. Then after 30 seconds, just as the game was ending, we regained control for a split second, then the match ended. Something smells fishy here... this happened multiple times, not just this once.
At the end of Friday, we ran a match with us on the stand. No one else was there, just Fusion and some of the FTA. Crowd was gone, the rest of the people at the FTA Table were gone. Here is what that match looked like:
http://i.imgur.com/cDvGql.png (http://imgur.com/cDvGq)
The code running at this time was our stripped code, but with CAN implemented still. Notice before they ran us, we were connected for a long time while Steve and the FTA guy discussed the situation. Never once did we lose a connection, and we had very low packet loss. We started the match in autonomous, CPU came up as expected, and we transitioned to teleop without any problems and ran until our chain came off, when the voltage became stable. Not once did we lose a connection after the match started. Now let's go to the next morning, first match of the day. Nothing on the robot had changed.
http://i.imgur.com/HwN4Ll.png (http://imgur.com/HwN4L)
Continued below...
Continued from above...
We drop packets, lose communication multiple times. Shouldn't this be enough to prove our point that it was not us? It ran fine with no one else there, but didn't run at all when people were there? And again, we were never offered the alternative router to use, and when Brandon Higgs asked, we were denied use of it.
Here's the second match of the day. Same code, but now we're using the classmate.
http://i.imgur.com/3bnMWl.png (http://imgur.com/3bnMW)http://i.imgur.com/Ea23Nl.png (http://imgur.com/Ea23N)
Nothing had changed. When we missed those balls during autonomous, it was because we lost communication with the robot... We maintained teleoperated for approximately 15 seconds before we dropped. Later in the match, we regained control for a few seconds. Then the match ended. After the match ended, we lost communication for a few seconds, then regained communication.
Our third match Saturday was bypassed due to an electrical problem we created between match 2 and 3 while replacing our 12V-5V converter. The router was never powered on, and we were bypassed that match.
Here was our fourth and final match of the day. CAN has been taken out at this point. We're running basically bare stock code. No camera, no CAN, no closed loop control. We had kit-bot programming in our Mercedes of a robot.
http://i.imgur.com/VZzEml.png (http://imgur.com/VZzEm)
We still dropped communication like crazy and didn't maintain control during the match.
Finally, we ran on the practice field in front of the FTA guy that was less than helpful to us for 20 minutes, scored 30/32 balls, and had two small glitches, but they lasted no longer than a fraction of a second and we regained control. We places our 'Mercedes' code back on it, and used the Dell again.
http://i.imgur.com/k9LoYl.png (http://imgur.com/k9LoY)
We're still reviewing data and trying to piece things together.
With all the reported trouble with this issue, including our own, there seems to be a problem out there. I have not seen much discussion of results with the rev. B router in competition. Can anyone report the same trouble with a rev. B router? We will be running a rev.B 1522 at St. Louis this weekend (wk 4), so we will report on what we learn.
Very slim chance considering FMS uses WEP. Thats the purpose of the radio kiosk that all teams must use to re-program their radios at each event. The WEP keys are randomly generated by FMS at each event so you team's key will change between each event. The WEP key list is not available to teams.
We've considered it. I'm curious as to the WEP key list. You said it's generated at each event, so it's not recreated each time you reprogram the router.
I talked with the student that reprogrammed our router. Apparently all you have to do is enter in your team number, and that's it?
So another team could simply bring their spare router, tell the kiosk that they're #364, and they would have our encryption key? Messing us up would be as easy as leaving the router powered on in the pit under some jackets or something, right? That would explain why it would have worked after everyone left on Friday. Power to the pits I think was already cut off. It wouldn't jam our signal on the practice field because we're using their network equipment.
I don't want to believe it is malicious, but it's hard to brush off. FTA was completely unwilling to help us out in regards of getting the spare router, and we even asked for a different (temporary) team number, and they refused to do that for us.
Kevin Sevcik
19-03-2012, 14:14
Ryan,
First, it's a WPA key. Second, the kiosk only sets up the radio for bridge mode. I don't have one here to verify, but I'm pretty sure switching over to AP mode wouldn't end up with a fully configured AP that your bridge would try to connect to. I could be wrong on that point, though. An identical radio in bridge mode shouldn't cause problems unless it's hooked up to something with a conflicting IP or something. At any rate, I'm really doubtful on that one. It just seems so unlikely that someone would go to that trouble. Incompetence seems unlikely as well, as who would accidentally program a radio with the wrong number and then leave it plugged in? Still, the self-serve WPA kiosk is definitely a gaping security hole in the system. I wouldn't advocate going back to the bad old days of handing out and programming WPA keys by hand, but handing out a team specific password for the kiosk would be a good idea.
Lastly, when you come to Lone Star, would you consider leaving everything as is for a practice match? I'm pretty sure Lone Star isn't getting the Bayou truck, so it'd be a pretty compelling data point if your robot, as is, does/doesn't work on the Lone Star field.
Ryan,
First, it's a WPA key. Second, the kiosk only sets up the radio for bridge mode. I don't have one here to verify, but I'm pretty sure switching over to AP mode wouldn't end up with a fully configured AP that your bridge would try to connect to. I could be wrong on that point, though. An identical radio in bridge mode shouldn't cause problems unless it's hooked up to something with a conflicting IP or something. At any rate, I'm really doubtful on that one. It just seems so unlikely that someone would go to that trouble. Incompetence seems unlikely as well, as who would accidentally program a radio with the wrong number and then leave it plugged in? Still, the self-serve WPA kiosk is definitely a gaping security hole in the system. I wouldn't advocate going back to the bad old days of handing out and programming WPA keys by hand, but handing out a team specific password for the kiosk would be a good idea.
Lastly, when you come to Lone Star, would you consider leaving everything as is for a practice match? I'm pretty sure Lone Star isn't getting the Bayou truck, so it'd be a pretty compelling data point if your robot, as is, does/doesn't work on the Lone Star field.
I'm doubtful on it as well. We're planning on leaving everything the same until we know what we're doing. We're not going to unbag it until we know if we're not going to Lone Star.
I just spoke with Greg on the phone for a nearly an hour. I sent him the email I sent above, and talked with him about all what was going on.
He doesn't have much to say yet. He is not sure what is causing the packet loss right before autonomous, or the short chunks of data that are going missing, as it's not enough time to mean a power problem. He said a bad ethernet problem could cause it. I don't believe that to be our case as we did pretty extensive testing to be sure that we were electrically good.
I'd really like to see FRC return to a 900MHz radio solution.
Consumer-electronics grade Wifi is just too widespread to count on there being little-to-no-interference.
We're all carrying in literally hundreds of 802.11 wifi-enabled devices that talk in the 2.4GHz and 5GHz bands, never mind that most of the venues have at least one of their own.
Captaindan
19-03-2012, 19:46
Can cell phones really use the bandwith on the field? (Via the the request for network usage) Last year we were told to shut off wireless devices. Now more then ever with cellphone usage up tenfold from last year is this the problem? Were we the the unlucky team with a 2.4 or 5 mhz bandwith?
We're still reviewing data and trying to piece things together.
Did you see this post in 330's thread?
I was FTAA at the regional at Utah this last week, and we were having similar problems. The FTA found that every team on the field had high trip times (>30ms) when there was a team with a D-Link running firmware version 1.4.
We has the CSA downgrade those bridges to v1.2 of the firmware and all of the trip times went back down to normal. I am not sure if this is the same problem that everyone else was having, but downgrading the firmware on any of the older bridges running 1.4 seemed to help.
-Clint
Just to throw that out there.
Dusk Star
19-03-2012, 22:10
Very slim chance considering FMS uses WPA. Thats the purpose of the radio kiosk that all teams must use to re-program their radios at each event. The WPA keys are randomly generated by FMS at each event so you team's key will change between each event. The WPA key list is not available to teams.
The fact that it is WEP does NOT make it secure... in fact, some people joke that it is easier to crack WEP at someone's houls (and faster) than to get up and ask them. All commercial wifi setups have flaws. For example, I have seen backtrack 5 crack my home wifi (linksys a/b/g router, set to WEP temporarily) in under two hours. Via a laptop. With WPA2, it took 4 (used the WiFi protected setup vulnerability). A sufficiently determined team COULD crack the encryption in a relatively short timeframe. Or, they could use any of the other specialized linux penetration testing distros. And if the WPA list key is not avaliable to teams, how hard would it be to imitate the FMS setup? (change MAC address on router, set up a network, log attempted connections and passwords)
Yes, I seem to have put too much thought into this. I do not intend to be the kind of person who would do this, however.
abbyjcox
19-03-2012, 23:10
Whatever the problem is....I just hope that it is fixed IF/when we go to Lonestar...I know that EVERYONE on the team worked our butts off to build what we all agree is the best robot we have ever made.. and it stinks beyond belief to see it all go to waste over a problem that wasn't even our fault.. Personally I am kinda disappointed with FIRST for how unhelpful they were, they above all others should be the prime example of gracious professionalism and this time they fell short.. However, I am very proud of my team mates and how they conducted themselves through all of our issues and I know that even if we are not able to compete and put all of our hard work to good use, that all that hard work was not completely wasted because we all learned A LOT this year, and still had fun in the process despite the hard time we encountered at the regional.
Alan Anderson
20-03-2012, 01:14
A sufficiently determined team COULD crack the encryption in a relatively short timeframe. Or, they could use any of the other specialized linux penetration testing distros.
You might find it enlightening to consider the total amount of time a given team's wireless network is even active at a regional.
You might find it enlightening to consider the total amount of time a given team's wireless network is even active at a regional.
And again, as I understand it...
Why try to crack the WPA or WEP, whatever it is, encryption when you can simply reprogram your router with another teams number?
wireties
20-03-2012, 09:40
And again, as I understand it...
Why try to crack the WPA or WEP, whatever it is, encryption when you can simply reprogram your router with another teams number?
But the FMS would detect duplicate IP addresses because I doubt the install script FIRST uses changes the machine address. So if something nefarious happened, I doubt it involved re-programming a router with the same team number (by FIRST) - which means they would have to hack comms to get the WPA keys. And a MITM attack seems too far-fetched for FIRST students to pull off.
Nope, there has to be another reason. We were a few spots down from you in the pits. I wish I new you were having trouble, we could have swapped some parts around and eliminated some causes. It is so unfair that you had an undiagnosable problem like that! Any luck with Greg?
Greg McKaskle
20-03-2012, 10:08
Greg is digging through the data I do have and I'm looking for more. I'm waiting on log files from another team at Bayou. If you have access to the DS computer, I'd really like to compare aspects of the files between teams with no perceivable issues and ones with issues. PM me if you are able to assist.
I'm also collecting log files as I force specific types of failures. I'm not sure if or when it will be known what caused the failures, but I'm learning of other things that would be nice to log and ways to better diagnose if this happens again.
Greg McKaskle
James Tonthat
20-03-2012, 10:14
Greg, forgive my ignorance, but I'll be telling my programming team about this.
Are there some instructions or something I can look up to help them find the log files?
Command67
20-03-2012, 10:59
I don't think the firmware's the problem. When we looked on Saturday at the router's information during regional it said v.1.21.
mcristina444
20-03-2012, 11:21
We attended the Bayou Regional and almost every time we we connected to Blue 3 we lost connection. The computer said we were connected to the FMS but the communication and robot code lights were off. The RSL never stopped blinking. Also once we were connected to Blue 1 and headed in front of the Blue 3 side ( by the ref table) and lost connection. After speaking with officals they stated that our battery dropped below 8 volts before dropping out, but the battery level on the dashboard was steady. Just my 2 cents.
2078 Driver
Greg, forgive my ignorance, but I'll be telling my programming team about this.
Are there some instructions or something I can look up to help them find the log files?
The log files are located in:
C:\Users\Public\Documents\FRC\Log Files
You can view them using the Driver Station Log File Viewer located in:
64-bit Operating System
C:\Program Files (x86)\FRC Driver Station
or
32-bit Operating System
C:\Program Files\FRC Driver Station
EricS-Team180
20-03-2012, 12:07
We attended the Bayou Regional and almost every time we we connected to Blue 3 we lost connection. The computer said we were connected to the FMS but the communication and robot code lights were off. The RSL never stopped blinking. Also once we were connected to Blue 1 and headed in front of the Blue 3 side ( by the ref table) and lost connection. After speaking with officals they stated that our battery dropped below 8 volts before dropping out, but the battery level on the dashboard was steady. Just my 2 cents.
2078 Driver
...hmmmm
George Wallace (SPAM'er, Exploding Bacon and Orlando emcee) said something interesting last night at our SPAM meeting. He was telling me that teams were reporting a dead spot on the field (Orlando regional): blue zone, near the scorer's table. Our freshman IT whiz-kid was telling me that, when we walked down to the field for the Finalist medals he took a peak at the field hardware. Apparently there is a cluster of high end routers located at the scorer's table in that corner. I was just talking to one of our CE/EE's, here at Belcan, who did a lot of work in the RF lab at FAU. Locating a group of routers in a cluster can and will cause near-field cancellation.
... yet another tidbit ...
Eric
...hmmmm
George Wallace (SPAM'er, Exploding Bacon and Orlando emcee) said something interesting last night at our SPAM meeting. He was telling me that teams were reporting a dead spot on the field (Orlando regional): blue zone, near the scorer's table. Our freshman IT whiz-kid was telling me that, when we walked down to the field for the Finalist medals he took a peak at the field hardware. Apparently there is a cluster of high end routers located at the scorer's table in that corner. I was just talking to one of our CE/EE's, here at Belcan, who did a lot of work in the RF lab at FAU. Locating a group of routers in a cluster can and will cause near-field cancellation.
... yet another tidbit ...
Eric
It's a single AP with 6 antennas (if you look closely, there are 3 for 2.4ghz and 3 for 5ghz).
In the range we are talking about (no more than 50' with no solid obstructions), 802.11 should have no problems with signal strength assuming the bridge is mounted well.
Gary Dillard
20-03-2012, 12:31
...802.11 should have no problems with signal strength
Seems to be the answer everyone is getting from the FTA - there shouldn't be a problem, therefore it isn't a problem, therefore it's your problem.
I guess we should just ignore all the data points that multiple, experienced, highly competitive teams are providing, since there shouldn't be a problem.
Seems to be the answer everyone is getting from the FTA - there shouldn't be a problem, therefore it isn't a problem, therefore it's your problem.
I guess we should just ignore all the data points that multiple, experienced, highly competitive teams are providing, since there shouldn't be a problem.
I've seen this thread mention many reasons other than 802.11 signal strength as the cause of issues. I only mentioned signal strength, in relation to a post questioning possible issues related to the design of the field AP.
Gary Dillard
20-03-2012, 14:01
I wasn't intending to pick on you Andrew; your post actually included good information related to the construction of the hardware. But it still highlights the issue that "there shouldn't be a problem with signal strength", yet as Eric said, multiple teams were reporting a "dead spot" on the field and mchristina said the same thing about a different regional, so the data says there is some kind of real problem that smells an awful lot like signal strength in some part of the field.
EricS-Team180
20-03-2012, 17:08
After mchristina444's post, it reminded me of what George had told me. So I just wanted to throw it out there so others more knowledgeable could add credence or call shenanigans. So thanks to apalrd for that :) Yet, let me ask a different way: Is it possible for there to be wave cancellations in the near field? ...some odd RF effect? Just curious.
I'm with Gary, though. We're not here to pick on anyone, but there is a body of evidence collecting here. I know people like Greg M. are following up and I am grateful for that. Let's see where the data leads.
Eric
Mark McLeod
20-03-2012, 18:29
... there is a body of evidence collecting here.
There's a body of anecdotal randomness...
Engineers measure things...
mcristina444
20-03-2012, 18:34
Also, after talking to multiple teams at the Bayou Regional, there was more than just my loss of signal by the scores table. Sadly I don't remember specific team numbers. Talking to the FMS representative, I repeatedly used the word "dead zone" because that was my first thought on the field. He shot down the idea immediately. Having the issue in that one area of the spot. He said "Coincidences happen more often than you think".
But one thing did confuse me, why was this only happening to ~5 teams if it was a dead zone in the wifi? Router placement on the robot? etc?
I'm not going to flat out blame this on the FMS but I do believe it is something to look into.
Having this happen more than 3 times in different matches can really bring down your rankings. I experienced that first hand.
2078 Driver
Jimmy the Kidd
20-03-2012, 19:34
The biggest idea I have is some sort of interference on the FMS or the 10.3.yy IP or something of that sort. The biggest evidence is the occurance of our problems only when there was a large crowd or devices. What else could the packet loss come from, since it only happens when there's a crowd?
James Tonthat
20-03-2012, 19:38
I've asked my team to pull our logs. We didn't have any comm issues during the Friday/Saturday at Bayou. The only problem we had in a quali match was traced back to controller issues in our second match.
Hopefully this helps.
Kevin Sevcik
20-03-2012, 19:51
I'll pull logs from our DS. We had control issues in two early matches due to a bad CAN network, and at the start of one match I saw our robot connect, run fine for a few seconds, lose comms, reconnect, lose comms and finally reconnect for good. There was one match in elims where we took FOREVER to connect and couldn't align our robot. Much longer than any other bot on the field. This despite the fact the we power up our robot shortly after the previous match ends. We had good comms during all of our matches as far as I know, however, despite simultaneous camera feeds going on the eeePC we use for a DS.
wireties
20-03-2012, 20:04
There's a body of anecdotal randomness...
Engineers measure things...
So true - I hate what happened to the teams in this thread but there is no evidence (yet). Let's say the FMS said "OK, this is our fault". What direct action (not redesign the system) could they take to fix it? We can't tell them what to fix!
DestinyKatT
20-03-2012, 20:55
Opportunity had our routers freak out and die. the one in the kit had the issue and it later litterally fried, then we got a new one and after a while of practicing and by the time we got to competition it started doing it so bad it was every match. it worked in the pit but not well on the feild. when we went to the officials spare parts station and got one lended to us, it worked perfectly, a couple other teams had issues with them too, i think it happened to the last few to hook up connections, or it was that a while after use the parts glitched up badly. Our advice is to bring a brand new one to competitions, and maybe a spare if affordable. i did notice that some of the last few teams to hook up wireless connection had issues staying online and connected on the feild. same with the practice feild
abbyjcox
20-03-2012, 20:58
Opportunity had our routers freak out and die. the one in the kit had the issue and it later litterally fried, then we got a new one and after a while of practicing and by the time we got to competition it started doing it so bad it was every match. when we went to the officials spare parts station and got one lended to us, it worked perfectly, a couple other teams had issues with them too, i think it happened to the last few to hook up connections, or it was that a while after use the parts glitched up badly. Our advice is to bring a brand new one to competitions, and maybe a spare if affordable
That wasnt our problem...
Captaindan
20-03-2012, 21:31
Opportunity had our routers freak out and die. the one in the kit had the issue and it later litterally fried, then we got a new one and after a while of practicing and by the time we got to competition it started doing it so bad it was every match. it worked in the pit but not well on the feild. when we went to the officials spare parts station and got one lended to us, it worked perfectly, a couple other teams had issues with them too, i think it happened to the last few to hook up connections, or it was that a while after use the parts glitched up badly. Our advice is to bring a brand new one to competitions, and maybe a spare if affordable. i did notice that some of the last few teams to hook up wireless connection had issues staying online and connected on the feild. same with the practice feild
Our router all connections and everything to do with the control system of the robot was replaced atleast twice the issue was never resolved we checked all code and various settings finally we went to data collecting because we had no support or solution from fta that we hadn't done The robot ran wirelessly on the practice field but did not connect reliably to the field
By the support of the FIRST community, Gracious Professionalism of FIRST, and the gracious support of Lucien Junken from Robonauts, Team 118, Team Fusion 364 is officially going to the Lone Star Regional!
Some things to mention at this point. When we get to the competition on Thursday, we will be going out to the field early along with some other problematic teams, and test communications. We're not going to change anything with our configuration, including software or hardware. We want to isolate the problem first, then go about some troubleshooting.
At this point, we are leaning towards RF interference. Our router was set to channel 1, but I'm not sure the channel configuration of other robots.
I know channels 1, 6, and 11 are most often used because their RF frequencies do not overlap.
Even though not all teams have problems, FIRST does have a problem.
Take for example this video from the iPhone 4 unveiling a few years ago:
http://www.youtube.com/watch?v=znxQOPFg2mo
The audience was way smaller. The iPhone 4 could not connect at all. To fix this problem, Steve Jobs had to ask the audience to shut off all electronic devices.
FIRST needs to do the same, or use a different technology. Simply using the 5GHz band of Wireless N would help tremendously.
This doesn't answer the question of why just us, but this is a problem at the competition.
For now, FIRST should ask the audience to place their cell phones into airline mode. Students on the field need to just leave their phone in the pit with them.
linuxboy
20-03-2012, 22:11
I'm not sure how the crowd's devices would have affected it (I'm not saying they didn't), but I don't think they could have caused an IP conflict. The 10.0.0.0/8 subnet is fairly uncommon for your average user, and they would need to somehow be on the network (the chance of their phone randomly being connected to your SSID with its encryption key are next to none). The only way that there could have been an IP conflict is malicious intent (and I'll talk about the issues with this momentarily). I don't know a lot (nothing really) about RF, but, I suppose that a lot of phones trying to scan might cause interference, but I would think it would be on the 2.4GHz spectrum (which I think the field doesn't even use here in the states, iirc, the 2.4GHz antennas are disabled, but must be inserted anyway). In terms of localized cancellation, I believe the field AP is rated for 16 networks, and is only broadcasting 7, so I don't think that should be a problem.
About malicious intent, the field uses WPA, as does the practice field, WEP is a broken encryption scheme, and is not used by FIRST. The actual field also probably does not have WPS (it isn't a home router after all). Someone talked about just programming a bridge as another team, it would work, but I feel like the problem would be a little less consistent (IP conflicts are strange beasts). That said, MITM is not out of the range of skills of FIRST students, after all, we all know how to use google, but setting up a MITM attack that would survive lots of network disconnects and reconnects would be fairly complicated (ettercap might be able to do it maybe, but I doubt it would work without significant professional training).
I tend to agree that "It should work, therefore it does" however I am aware that the field doesn't always work, and I really hope that you find the source of your issue, I for one will be watching this to see what the result is.
Good Luck,
Oliver
AllenGregoryIV
20-03-2012, 22:12
Has anyone had success with other USB IO Modules? I know there was one which emulated a joystick (can't remember the name/seller) posted earlier in the season from a smaller robotics site.
I think you are thinking off the Custom Control Interface (http://www.estoprobotics.com/estore/index.php?_a=viewProd&productId=33)from eStopRobotics. I haven't gotten a chance to use it, so I can't say if it works well or not.
I've had bad experiences with the cypress board in the past and instead we choose to have a student write a custom program for the teensy (http://www.pjrc.com/teensy/)board. One of our programmers wrote the code to make it emulate a joystick. I don't have it to post yet but we should soon. He said it was only about 20 lines of code and most of the work was already done by other teensy developers. We haven't run this through a competition yet, we go to Dallas next week.
RufflesRidge
20-03-2012, 22:15
FIRST needs to do the same, or use a different technology. Simply using the 5GHz band of Wireless N would help tremendously.
They do. I have also seen an AirTight sensor on the scoring table, I was curious what it was so I did some googling: http://www.airtightnetworks.com/home/products/cloud-services/wireless-security.html
It sounds like you were trying to pull a channel setting off of the radio. he client does not choose the channel in a WiFi setup, the AP does meaning that either all teams or every team in a single station would likely use the same channel. Given Andrew's comment of only 3 antennas per frequency it is likely that all 6 SSIDs are broadcast on the same channel meaning all teams are using the same channel.
linuxboy
20-03-2012, 22:52
They do. I have also seen an AirTight sensor on the scoring table, I was curious what it was so I did some googling: http://www.airtightnetworks.com/home/products/cloud-services/wireless-security.html
It sounds like you were trying to pull a channel setting off of the radio. he client does not choose the channel in a WiFi setup, the AP does meaning that either all teams or every team in a single station would likely use the same channel. Given Andrew's comment of only 3 antennas per frequency it is likely that all 6 SSIDs are broadcast on the same channel meaning all teams are using the same channel.
There are a couple devices on the scoring table, the AP, is what broadcasts the networks, the Air Tight, is very good at scanning surrounding networks, and can even shut them down (FIRST hq must be consulted before that happens). I believe FIRST actually has an FCC license to operate those. The air tight may be used for other things, but thats what I've been told about it.
GrimmReaper
20-03-2012, 22:58
Yeah i'd like to comment on my understanding of the field WiFi setup and maybe this will help clear up a few concerns. I am not an expert on the FRC Field, but i have extensive networking knowledge, I have not been involved in the field setup, but i've done a lot of research on it and I do believe I have a good grasp on this, as such - if i am incorrect - please correct me ;-)
Think of it like this, if you walk into starbucks with your laptop and connect to their wifi, you don't choose the channel you are on - the starbucks router does. If you and 5 other buddies all walk into starbucks and connect - you're all on the same single router, on the same single channel all connecting to chiefdelphi at the same time with no issues. The field is essentially the same situation - your robot routers in bridge mode are equivalent to a laptop at starbucks - but they are really a middleman allowing you to connect both a camera and crio to that network - but they are essentially conneting the same way.
Specialties for the FRC Field
1) I believe in most cases the field WILL be configured for 5ghz, all of our routers are capable of 5ghz, the lack of 5ghz on classmates is irrelevant as they are physically connected for the matches.
2) I'm unsure if every team is given an individual WPA password via the kiosk or if they are all the same WPA password, however the kiosk does set the ip address of the router for sure. Separate passwords could allow them to restrict acces a little better, but it can still be done with a single shared key.
3) From what i've gathered - the field will only accept traffic from those teams that are scheduled on the field, so in theory any additional potential traffic from other teams in queue or elsewhere should not be present on the field network during a match.
4) all teams while connected to the field are essentially part of one big network, in theory you could all connect to each other, and you are all able to connect to the field FMS Server, kinect kiosks, etc... However this traffic COULD easily be limited on the networking side, but i'm unaware if the FRC Field network sets up anything to segment the team networks.
5) Someone mentioned multiple SSID's. It's possible that they actually setup separate SSID's for each team, but I doubt they'd set it up that way, i don't see a good reason for that, there is most likely one single SSID, which may not even be broadcasted, so that your average high school hacker would have a more difficult time trying to get into the field network. Presumably there is just one SSID - much like the one SSID you'd connect to at Starbucks...
If i left any details out, let me know... but i believe that's a fairly good representation of the field network.
As for the actual problems that teams experienced and possible causes, i wish i could add some insight to that, i've been racking my brain on it for the past couple of days, watching tirelessly for someone to come up with the magical answer so i can stop wondering as well :-)
Good luck to everyone!
- Jon
EricS-Team180
20-03-2012, 22:59
There's a body of anecdotal randomness...
Engineers measure things...
Yep, the only plots of anything I've seen here are by RyanN. Is that data good enough to find a root cause? Like I said, "Let's see where it leads."
Do you have any suggestions of what we can measure Mark?
Eric
RufflesRidge
20-03-2012, 23:09
5) Someone mentioned multiple SSID's. It's possible that they actually setup separate SSID's for each team, but I doubt they'd set it up that way, i don't see a good reason for that, there is most likely one single SSID, which may not even be broadcasted, so that your average high school hacker would have a more difficult time trying to get into the field network. Presumably there is just one SSID - much like the one SSID you'd connect to at Starbucks...
There's an SSID for each team on the field set to the team number (e.g. "2046"). You can see this with a laptop with 5GHz capabilities, or by logging into your bridge and looking at the settings.
linuxboy
20-03-2012, 23:16
Yeah i'd like to comment on my understanding of the field WiFi setup and maybe this will help clear up a few concerns. I am not an expert on the FRC Field, but i have extensive networking knowledge, I have not been involved in the field setup, but i've done a lot of research on it and I do believe I have a good grasp on this, as such - if i am incorrect - please correct me ;-)
Think of it like this, if you walk into starbucks with your laptop and connect to their wifi, you don't choose the channel you are on - the starbucks router does. If you and 5 other buddies all walk into starbucks and connect - you're all on the same single router, on the same single channel all connecting to chiefdelphi at the same time with no issues. The field is essentially the same situation - your robot routers in bridge mode are equivalent to a laptop at starbucks - but they are really a middleman allowing you to connect both a camera and crio to that network - but they are essentially conneting the same way.
Specialties for the FRC Field
1) I believe in most cases the field WILL be configured for 5ghz, all of our routers are capable of 5ghz, the lack of 5ghz on classmates is irrelevant as they are physically connected for the matches.
2) I'm unsure if every team is given an individual WPA password via the kiosk or if they are all the same WPA password, however the kiosk does set the ip address of the router for sure. Separate passwords could allow them to restrict acces a little better, but it can still be done with a single shared key.
3) From what i've gathered - the field will only accept traffic from those teams that are scheduled on the field, so in theory any additional potential traffic from other teams in queue or elsewhere should not be present on the field network during a match.
4) all teams while connected to the field are essentially part of one big network, in theory you could all connect to each other, and you are all able to connect to the field FMS Server, kinect kiosks, etc... However this traffic COULD easily be limited on the networking side, but i'm unaware if the FRC Field network sets up anything to segment the team networks.
5) Someone mentioned multiple SSID's. It's possible that they actually setup separate SSID's for each team, but I doubt they'd set it up that way, i don't see a good reason for that, there is most likely one single SSID, which may not even be broadcasted, so that your average high school hacker would have a more difficult time trying to get into the field network. Presumably there is just one SSID - much like the one SSID you'd connect to at Starbucks...
If i left any details out, let me know... but i believe that's a fairly good representation of the field network.
As for the actual problems that teams experienced and possible causes, i wish i could add some insight to that, i've been racking my brain on it for the past couple of days, watching tirelessly for someone to come up with the magical answer so i can stop wondering as well :-)
Good luck to everyone!
- Jon
I don't mean to be mean, however some of that is incorrect I will go through the items and comment about my understanding.
1) I agree
2) Each team has a separate WPA key, they are generated Wednesday night I believe
3) This is done by having a separate SSIDs (unbroadcast I _think_, but the same as the team number I would guess) and keys for each team
4) Each team has it's own VLAN, I'm not too good at VLANs so I can't describe this too well, FMS can talk to anything, but each DS cannot talk to other DSs or other Robots (this is why, for those of you that have done field setup, each player station has a labeled port on the SCC, instead of just plugging into the switch)
5) The field AP is reconfigured every match, with the SSIDs for the teams in that match, with the appropriate WPA keys. This is why the scorekeepers must "prestart" the match. The SSIDs are not broadcast I _think_
What you described seems almost like it could be a FMS Light setup for a competition.
I know these things from doing quite a bit of volunteering, and asking the FTA a lot of questions while I do.
I agree, though with that last bit.
- Oliver
Yep, the only plots of anything I've seen here are by RyanN. Is that data good enough to find a root cause? Like I said, "Let's see where it leads."
Do you have any suggestions of what we can measure Mark?
Eric
I have all of Team Combustion's logs from Bayou. I have close relations with them since one of their mentors was my mentor at the NASA Stennis Space Center for four internships. We're a sister team of theirs.
I'm focusing (read: trying to focus) on my Advanced Circuits exam I have tomorrow morning at 8AM. After the exam, I'll review their data, try to match up some data with ours and do some comparisons. I'm really interested in seeing what the data will show.
I have send the logs to Greg as well.
I'm still sticking with too much WiFi traffic and interference for our connection problem until I figure out otherwise.
Gary Dillard
21-03-2012, 08:13
There's a body of anecdotal randomness...
Engineers measure things...
I'd have to disagree with you semantically here Mark. If team A loses comms or their cRIO resets during match 1 at a certain location, that's a data point. If they don't during match 2 at the same location, that's another data point. If team B loses comms at a different location, that's another data point..... specific events and frequency of occurence is data. Pass/fail tests are data as well - it always worked when we did this. The problem is, engineers tend to ignore things that can't be quantified (qualitative data). If a jet engine on the test stand starts spitting parts out the back, it doesn't really matter whether the sensors indicated something went wrong or not - the evidence speaks for itself.
The FMS' job is to communicate with the robots, and it isn't happening hundreds or thousands of times per season. Are the vast majority of the problems robot issues? Probably, but multiple teams have done extensive troubleshooting and don't have issues when they're not running through the FMS. If it's performing as designed, then it's designed wrong. If it has some limitations, either they are not being communicated well to the teams or no one is interested in looking for them.
Greg McKaskle
21-03-2012, 09:39
I think Mark's point is that measurements result in data, numbers, etc. They are independently verifiable and repeatable. Anecdotal data is valuable too, but you need to apply valid statistics to it and work hard to remove bias. Engineers act on hunches and anecdotal data all the time, but that doesn't make the data they acted on to be measurements.
If a team member walks to the FTA and points out that five of the field lights are not working, that is an easily verifiable, quantitative measurement. We all have sensors/eyes for making that measurement. We also have experience helping us to evaluate the impact -- are the clustered or spread, where are they pointing, do we have spares, which should we replace to make the most impact, etc.
If humans had RF receptors, comments about a dead spot would probably be acted upon similarly. But since we don't, I'd hope the FTA would scale the reports by a credibility factor and file it into the "something to watch for" bin. I suspect that if they also saw consistent secondary evidence of a dead spot, they would call Manchester and see how they can make a hard measurement, or verify their hunch, or switch to an alternate setup recommended by FIRST.
RF is complex and not intuitive to those not specifically trained in it -- including myself. Wifi builds on RF, and adds its own 30 years of protocol complexity. The robots have their complexity as well, and as I've said before, the majority of the time, the issues can be demonstrated to be a robot or DS issue.
In the best cases, lets call them category-one, we can make primary and repeatable measurements that everyone can learn from. We can measure the battery to determine if it was at fault. We can jiggle wires and show a particular wire that causes the issue -- because I can reproducibly cause the fault and we can all see it. We could also scope it if necessary and measure the circuit fault and give numerical data to the circuit fault, but that usually isn't necessary. Problems like these result in good learning opportunities.
Other times, category-two, we can only eliminate and verify that common issues aren't present. We replace components until the lack of a fault gives us a likelihood that we have identified the issue. These are far less satisfying and prone to misinterpretation. Often we replace components and the issues doesn't go away. Does that mean the component that was removed was good, or simply that it wasn't the only bad component. In a system where multiple elements can fail and the symptoms look similar, this can be frustrating, and it is hard to make it into a good learning opportunity. Whenever possible, direct and independent measurements of the simplified system will help turn this into the former case where we are all learning more and more satisfied with the outcome.
I believe 364 did a very good job trying to troubleshoot their situation. I wasn't at the event, but I believe that both they AND the FTA and FIRST staff were drawing conclusions based on an educated interpretation of sketchy data. I am trying to be objective and glean what I can from the data. I am reviewing their logs, the logs of other teams at the event, and making my own logs of known failure points to try and identify a set of likely causes and eliminate others. If nothing else, I would like to put the right tools in place so that more of the category-two issues turn into category-one issues. If other teams have logs they would like to send me, PM and I'll give you an email destination.
Greg McKaskle
Dusk Star
21-03-2012, 09:58
3) This is done by having a separate SSIDs (unbroadcast I _think_, but the same as the team number I would guess) and keys for each team
- Oliver
I can confirm that the SSIDs are broadcast, and that they are named with the team number; the first competition I went to this year with my laptop I looked at the wifi networks and immediately went "someone is going to get yelled at for connecting over wifi..." After a match or two, I saw that the SSIDs were those of the ones currently in a match, and started to ignore it. I don't know whether they were 5ghz though, as I wasn't about to open a wifi diagnostics tool just to see what network they were on. I do know I have both frequency sets, though- 3x3 wireless a/b/g/n card
GrimmReaper
21-03-2012, 13:41
I don't mean to be mean, however some of that is incorrect
You're not being mean, I asked you to correct me if i was wrong :-)
I'm just happy to have more detailed information about the FMS specifics, thank you for that. It's actually architected a bit better than I expected which is good to hear ;-) Thanks to everyone that helped clear up my incorrect details.
So just to clarify my points from earlier on the channels - we still have no control over channels, that is going to be assigned by the FMS router, however if we are all on different SSID's they could be all the same channel or all on different channels. If they're on different channels, how do they allocate the channels? at random during pre-start? are they pre-set for each station i.e. Red-1 is always 5ghz channel 11, or something like that? I still don't see how that could cause any of the issues 364 saw, unless they were always on the same channel and it was a noisy channel... It would be good to know how they allocate channels.
- Jon
Ryan,
Thank you very much for the kind acknowledgement BUT, for the record, I was only one of very, very many that helped get the ball rolling for Fusion to have an opportunity to compete at Lone Star. With that said, it is a pleasure helping a team with such a rich history of helping others.
----------------------------------------
Everyone,
For the most part, this has been a good discussion along with good input BUT we still don't have a good answer for what has happened much less a root cause for 364's problems. The Lone Star Regional is very interested in helping 364 and FIRST determine the root cause of these problems. In this spirit, please keep posting ideas of what we should do at the Lone Star Regional to help troubleshoot this problem; we will develop a troubleshooting flow based on everyone's input.
Unless we get a better idea, our first test will be to unbag 364's robot on Thursday and attempt to hook it to the field (e.g. take robot out of bag, put battery in robot, get WPA key, etc.). We also intend to get other robots that have successfully connected at other regionals (57, 118, 231, 418, 624, 653, 1429, 1477, 2415, 3335, etc.) to test their field connections and so we can also have 6 robots connected at the same time. Depending on these test results, we probably have hundreds of options for what the next test should be; if you have any ideas, we'd like to hear about them and we don't care how silly they are (e.g. if 364's robot has no problems connecting, ask everyone in the arena to start turning on the wireless on their laptops and call their mothers from their cell phones to see if we can get 364's robot to shut down ..... if 364's robot will not connect, reconfigure 118's robot to look like 364's robot, use 364's bridge and see if 118's robot connects - the regional may even supply a "Basic Bot" that we can reconfigure). I will guarantee you one thing, our FTA will take everyone's input and will have a rock-solid troubleshooting flow when he walks in the door on Holy Thursday.
Thanks & give us your troubleshooting ideas,
Lucien
Tom Bottiglieri
22-03-2012, 03:55
Thanks & give us your troubleshooting ideas,
Lucien
Knowing what part of the system is broken quickly should help.
I would run tcpdump at three different places and compare the captures to find where packets are getting held up or lost. Run one on the DS (or on the same switch), one over the air (using the team's SSID and PSK), and one behind the robot bridge. You can probably use one laptop to do both of the latter.
Basically what you are looking to do is trace packet flow. Check the timestamps of packets as they move from the DS to the AP to the bridge to the bot. It will be pretty apparent if there is a faulty piece of hardware in the chain. (I'm not as worried about the Cisco AP as I am the Dlink bridge you can buy at Walmart...)
If packets aren't getting delayed or dropped anywhere in that chain, check to see how it takes for replies to be generated by both the DS and cRIO.
PS. You can use Wireshark instead of tcpdump for prettier output, or if you want a GUI.
This may not be 364's issue but I am wondering if we're just seeing the result of more 5Ghz 802.11n being rolled out to devices in the stands. At numerous regionals we get consistent reports of robots working fine on the practice field, in their labs and even on the competition field before or after the day is over but failing in competition.
At the Oregon regional my phone was showing dozens of active devices all broadcasting their whereabouts in the stands. Some of this might be teams setting up WiFi hotspots for competitive analysis (against the rules) but many were people oblivious to what their laptop or iPhone/Android was up to.
I'd suggest a first step for regionals this week is, when problems develop, have EVERYONE in the stands turn off WiFi on their laptops and phones and use a sniffer to see if that has taken place.
The other thing that has changed this year is more and more teams are streaming video from their robots at fairly high resolution (I know we are).
Jimmy the Kidd
22-03-2012, 19:22
This may not be 364's issue but I am wondering if we're just seeing the result of more 5Ghz 802.11n being rolled out to devices in the stands. At numerous regionals we get consistent reports of robots working fine on the practice field, in their labs and even on the competition field before or after the day is over but failing in competition.
At the Oregon regional my phone was showing dozens of active devices all broadcasting their whereabouts in the stands. Some of this might be teams setting up WiFi hotspots for competitive analysis (against the rules) but many were people oblivious to what their laptop or iPhone/Android was up to.
I'd suggest a first step for regionals this week is, when problems develop, have EVERYONE in the stands turn off WiFi on their laptops and phones and use a sniffer to see if that has taken place.
The other thing that has changed this year is more and more teams are streaming video from their robots at fairly high resolution (I know we are).
This is my exact line of thinking. I maybe looking into this too superficially, but when we don't work, it's when there are people in the crowd. Most of them are tech savvy, gadget toting people. Is it wrong to hypothesize that the root of our problem is as Dale says?
Is there an update on how 364 faired at connecting and working during practice day at Lone Star?
This is my exact line of thinking. I maybe looking into this too superficially, but when we don't work, it's when there are people in the crowd. Most of them are tech savvy, gadget toting people. Is it wrong to hypothesize that the root of our problem is as Dale says?
It isn't wrong to hypothesize, but don't take the hypothsis as the solution until you have checked the data to ensure all data is accounted for.
When 330 was experiencing the same symptoms in Los Angeles they continued even after almost everybody but the floor sweepers were out of the building.
To reiterate, the cRio rebooted itself at apparently random intervals while running default code with all circuit breakers removed from the Power Distribution board, the camera was also disconnected. No extraneous APs were observed or detected.
330's problem seems different than what we were seeing in Oregon. There radios were just disconnecting.
James Tonthat
22-03-2012, 20:43
Is there an update on how 364 faired at connecting and working during practice day at Lone Star?
Lone Star is week 6.
Ryan,
Thank you very much for the kind acknowledgement BUT, for the record, I was only one of very, very many that helped get the ball rolling for Fusion to have an opportunity to compete at Lone Star. With that said, it is a pleasure helping a team with such a rich history of helping others.
----------------------------------------
Everyone,
For the most part, this has been a good discussion along with good input BUT we still don't have a good answer for what has happened much less a root cause for 364's problems. The Lone Star Regional is very interested in helping 364 and FIRST determine the root cause of these problems. In this spirit, please keep posting ideas of what we should do at the Lone Star Regional to help troubleshoot this problem; we will develop a troubleshooting flow based on everyone's input.
Unless we get a better idea, our first test will be to unbag 364's robot on Thursday and attempt to hook it to the field (e.g. take robot out of bag, put battery in robot, get WPA key, etc.). We also intend to get other robots that have successfully connected at other regionals (57, 118, 231, 418, 624, 653, 1429, 1477, 2415, 3335, etc.) to test their field connections and so we can also have 6 robots connected at the same time. Depending on these test results, we probably have hundreds of options for what the next test should be; if you have any ideas, we'd like to hear about them and we don't care how silly they are (e.g. if 364's robot has no problems connecting, ask everyone in the arena to start turning on the wireless on their laptops and call their mothers from their cell phones to see if we can get 364's robot to shut down ..... if 364's robot will not connect, reconfigure 118's robot to look like 364's robot, use 364's bridge and see if 118's robot connects - the regional may even supply a "Basic Bot" that we can reconfigure). I will guarantee you one thing, our FTA will take everyone's input and will have a rock-solid troubleshooting flow when he walks in the door on Holy Thursday.
Thanks & give us your troubleshooting ideas,
Lucien
Try taking wireshark captures if possible with the field setup during the debug process for some post processing and clear evidence of the issue, if it's network related. http://www.wireshark.org/ and see this: http://wiki.wireshark.org/CaptureSetup/WLAN I haven't messed with wireshark in a while but I had to use it once for checking against some port scanning attacks on a network once, it's a pretty cool tool.
I would also try throwing on Tomato Firmware to some spare routers and set them up the same as they are in a normal match to see if you can get some more network analysis or if the firmware seems to fix the issue. http://en.wikibooks.org/wiki/Tomato_Firmware
Using either one of these utilities should make the issue stand out out like a sore thumb especially if it will be easily reproducible in Lonestar.
The last possibly could just be Ryan, I've never seen someone break so many macbook pro's in my life without touching any hardware but my boy RyanN can. I'm really just kidding on that aspect of course but seriously give some of these utilities a try.
-Brantley
Ah silly me, for a reason I thought it was this week. My mistake, time to wait out another week.
As a mere field reset volunteer I don't have many details, but St. Louis was having a ton of connectivity problems on Thursday... far more than I remember from the last couple years.
Andy Baker
23-03-2012, 08:55
As a few people mentioned before, one of your tests should include swapping out the PD Board.
Kudos to Lucien and all of the others who made this trip to Houston available for team 364.
Andy B.
Kris Verdeyen
23-03-2012, 11:32
As a mere field reset volunteer I don't have many details, but St. Louis was having a ton of connectivity problems on Thursday... far more than I remember from the last couple years.
Does anyone know where the field in St. Louis this week was last week?
BrandonHiggs
23-03-2012, 11:33
Changing the PD board is at the top of our list. It is the only piece of hardware that we did not replace at Bayou.
Changing the PD board is at the top of our list. It is the only piece of hardware that we did not replace at Bayou.
The only reason we didn't change it was because it is a pain to change, and it would have taken too much time.
If the problems continue at Lone Start, then it will be replaced.
Everyone,
Thank you very much for your ideas & input. We will be documenting all of our testing that we conduct at the Lone Star Regional and will share it with you here.
Let the Problem Solving begin,
Lucien
Akash Rastogi
05-04-2012, 03:47
Everyone,
Thank you very much for your ideas & input. We will be documenting all of our testing that we conduct at the Lone Star Regional and will share it with you here.
Let the Problem Solving begin,
Lucien
I think everyone wishes you the best of luck, also especially after what happened at the CT regional.
Best of luck 364 and all others who had FMS issues.
Test 1 completed: 364 successfully connected to the field and ran a match by themselves; similar to what 364 did Friday evening at Bayou with similar results.
... next test is to get other robots that send lots of data over the wireless connected with 364.
Wendy Holladay
05-04-2012, 15:37
what is the latest with fusion?
I just talked to RyanN on the phone. He said the robot is running great and they aren't seeing issues with the Lone Star field. Does anyone know where the Bayou field went?
Wendy Holladay
05-04-2012, 17:59
Chuck Kennedy at FIRST would know that
Everyone,
The executive summary with Fusion is, "All is well with 364 ... and we hope it stays this way." Fortunately and unfortunately, we found nothing wrong with 364's robot during our test runs. Here is the summary of test runs. A "(C)" denotes that the team was receiving camera video at the player station.
Test 1: 364 runs match by themself
Result 1: Normal operations
Test 2: R1-1429(C); R2-bypassed; R3-bypassed; B1-2936(C); B2-364(C); B3-118 (C)
Result 2: Normal operations
Test 3: R1-118(C); R2-1429(C); R3-bypassed; B1-364(C); B2-3847(C); B3-bypassed
Result 3: Normal operations
Test 4: R1-bypass; R2-118(C); R3-bypassed; B1-364(C); B2-3847(C); B3-bypassed
Result 4: 118 dies after being enabled for hybrid mode - Watchdog not fed - similar to what was seen in Connecticut; 364 & 3847 ran without issues
Test 5 (Same as Test 4): R1-bypass; R2-118(C); R3-bypassed; B1-364(C); B2-3847(C); B3-bypassed
Result 5: Normal operations
Test 6: R1-bypassed; R2-bypassed; R3-3103 (C); B1-364(C); B2-118(2xC); B3-418(C)
Result 6: Normal operations
Test 7: R1-2882(C); R2-364(C); R3-bypassed; B1-118(2xC); B2-647(C); B3-2585(C)
Result 7: Normal operations
At this point, we concluded the tests and declared 364's robot good-to-go. They have run a handful of times since the tests without issue; I'll let 364 update everyone with the specifics. The FTA, Jonathan, and his crew were awesome during all of these tests.
Finally, the engineer in me wanted to find a smoking gun with 364's robot but the other part of me is happy to hope that all will be well .... after all, it is the Easter season.
We'll keep everyone updated as the Lone Star Regional progresses,
Lucien
All is well with us.
We changed one thing. The router. During Bayou, we had tons of problems (obviously), so we used the pit admin spare. Firmware version unknown.
PD Board is still the same. We ran perfectly. I looked through some log files and saw no problems. I'll go through more tonight and report back.
Friday will be the day we would disconnect. Stands were basically empty. No interference.
The difference in the log files I looked are night and day difference from Bayou. We peaked out at 50ms latency at our highest point. Very little packet loss.
We're not holding our breath.
Thanks everyone that has helpe us out so far. We're crossing our fingers. We're here and ready to compete.
I'll keep everyone updated the best I can. I'm stuck with my mobile phone for the weekend.
-Ryan Nazaretian
Tom Line
05-04-2012, 22:07
Here's hoping you kick some butt and have a solid regional outting. Good luck.
JaneYoung
05-04-2012, 22:24
I just talked to RyanN on the phone. He said the robot is running great and they aren't seeing issues with the Lone Star field.
Ya'll should have seen some of the smiles on the team's faces when the robot was first being tested. It was pretty special to see.
All the best, Fusion.
Jane
GrimmReaper
05-04-2012, 22:34
I happened to be in Houston this week working, finished up today. So i plan on stopping by Lone Star to check things out Friday :-)
Good Luck!
So I browses through the log files and everything looks great.
Trip Time is 5ms on average. We have a consistent spike to 15ms every 30s or so. Non-issue. Assuming it to be a Windows service.
Weird phenomenon though. CPU usage continually increases. Usage starts around 40% and will raise to 45% throughout the match. It's non issue for a 2:30 match, but when practicing for 20 minutes, we can increase to 70% or so.
I wish I could post pictures, but that's a limit to the iPhone. :p
rachelholladay
05-04-2012, 23:12
To Ryan and all of 364: GOOD JOB. I hope its stays good all weekend.
We (Mum and I) were thinking about you guys all day.
To Ryan and all of 364: GOOD JOB. I hope its stays good all weekend.
We (Mum and I) were thinking about you guys all day.
I know. Won't me like 3 Facebook messages. I told her I have text messaging. :)
Does anyone know where the field in St. Louis this week was last week?
I was told it was from Bayou.
Jared Russell
06-04-2012, 08:15
Weird phenomenon though. CPU usage continually increases. Usage starts around 40% and will raise to 45% throughout the match. It's non issue for a 2:30 match, but when practicing for 20 minutes, we can increase to 70% or so.
This is interesting. What language are you using? If Java, I wonder if this is the garbage collector running. If C++, I don't have a good explanation (probably something in your teams' code). If Labview, I have no clue but suspect it might be memory management related (no idea how Labview actually does memory management under the hood).
EDIT: I assume you were referring to the cRIO's CPU usage. Is this correct?
This is interesting. What language are you using? If Java, I wonder if this is the garbage collector running. If C++, I don't have a good explanation (probably something in your teams' code). If Labview, I have no clue but suspect it might be memory management related (no idea how Labview actually does memory management under the hood).
EDIT: I assume you were referring to the cRIO's CPU usage. Is this correct?
It's LabVIEW, and CPU usage is on cRIO. I could understand if I was doing something with arrays and processing them, but I'm not.
We're about to have our first match. Fingers crossed. Everything on the field looks good so far.
Here is a play by play of Fusion in their first match:
Hooked to field
2 shots made in autonomy
alliance bridge to get balls
1 for 3
0 for 2
Coop with 624
...... 364 was fully operational
linuxboy
06-04-2012, 11:52
I watched the webcast to see you guys work!! I'm very happy that you're working, the engineer in me is a little sad because I want to know what the root cause of the problem was, but congrats on the co-op balance, and good luck in your future matches.
- Oliver
2nd match .... fully operational
To team 364: your robot is a very impressive and accurate shooter! I'm cheering for your team when I'm watching the webcast!:cool:
No issues all day. I'm too saddened we could not find the root cause. I'm also extremely curious.
Thanks again for all the help everyone has provided. I also want to thank all the volunteers at Lone Star that have helped us make it happen.
-Ryan Nazaretian
Jared Russell
06-04-2012, 20:22
No issues all day. I'm too saddened we could not find the root cause. I'm also extremely curious.
Thanks again for all the help everyone has provided. I also want to thank all the volunteers at Lone Star that have helped us make it happen.
-Ryan Nazaretian
Well, it is either the Bayou field/wireless environment, or your old radio, right?
If there is time after the event or during lunch tomorrow, maybe you can try the old radio?
Wendy Holladay
06-04-2012, 20:49
great day for fusion, mazel tov and enjoy lone star. still i too would like to really know what was the issue at bayou.......
smclean1969
10-04-2012, 12:09
Ryan
We had a similar problem at the Madera tournament last week. We had a single good match and then the robot started going off line. Out of the remaining 10 matches, we had a single match where we had full connectivity between our robot, driver's station and the field. Most of the matches, we never were able to communicate with the robot.
Similar to what you did, we completely tore apart the robot throughout the competition. We started by replacing the Dlink, but eventually replace the software, the power distribution board, the digital sidecar and the CRio. We actually went through four separate routers and three power distribution boards.
The robot started to come back at the end of the competition and we maintained full connectivity at the final match. The problem was, by the time we go to that match, we had rewired so much and some of the connections were done incorrectly that the robot failed to perform properly.
In hindsight, I'm pretty certain that the issue was with the CRio because once we replaced that, the robot started to recover. We replaced that as the very last item. I definitely wouldn't recommend swapping out as many electrical components as what we did unless it's an absolute necessity. I don't think software was ever an actual issue with the robot not performing correctly.
This was extremely difficult to try and lead the team through. At the same time, I'm immensely humbled and grateful to the mentors and students from other teams, members from the FIRST competition, National Instruments, CSAs and FTAs that tried to help us get the robot back on line. I think that's the great testimony to the people involved with the FIRST program that they provided this much assistance.
We'll keep working our issue and try to find the actual failure. We'll also have to spend some time analyzing our team processes that might have possibly caused that failure.
Scott
tagayoff
11-04-2012, 04:09
Ryan,
We just didn't have similar problems with what you experienced we had almost the exact problems you guys had. Even to the running all alone on an empty field Friday evening with no problems. Then coming back Saturday morning with the same problem. We had run only two practice matches on Thursday. The first with than 1522 still programmed from the LA regional so it just sat there with no connectivity for the first match. We reprogrammed it for the second practice session and the robot worked but was physically damage by a collision. I was not a witness to this match so I don't know the details. Our first match on Friday morning the robot started in teleop but lost communication almost intermediately for 42 seconds then lost again for another 40 seconds or so. End of match. Second time lost comm at the start of the match again the only movement I saw was our shooter motor trying to start then we lost comm. The FTA said it was for 40 seconds or so they suggested it was the Crio rebooting. I never saw the RSL light go off during any of the matches so I don't think the cRIO rebooted. I'm not sure what we replaced the next match and so on but we tried everything you did 3 radios , cRio twice, 12-5volt converter, PDB 3 times , basic source code, removed all other supposed power noise problems except the drive wheels. For our last 5 matches on Friday I watched from behind the FMS display. As soon as they reloaded the display with our match teams the display would show us as disconnected and dropping packets. They would wait till we intermittently showed green then start the match. Needless to say we lost comm every time at the start. Did I mention that Friday night when everybody was gone we ran OK? The order of replacing the different items are not clear to me as they weren't under my control. At this point I was just shaking my head and thinking how are we going to get this robot back together for St. Louis. On our 9th match we had replaced the digital side car?? and the cRio again along with the cards?? Now this is what I can say for certainty about this match. I'm sitting behind the FMS display again at the start of the match they bring our robot onto the field the display is updated to our current match. The display shows our robot dropped comm with packets dropping constantly. Two FTA's go to our robot do something (several minutes went by ) (queue dancing and loud music) They start to leave the arena and our robot shows connectivity and no dropped packets. Match starts our robot moves sort of Jags miss wired ball pick up set to the compressors pwm signal but we move the whole match with very few packets lost. I asked the FTA what he did and he said they connected the driver station to the robot when they saw no connectivity and it connected ok through a cable. But when he disconnected the Driver Station the radio would not connect to the field so he powered off the robot to rebooted the robot. When it came back up we have connectivity for the entire match. Next match we make sure we do not connect the robot to our driver station by cable or if we do we power the robot off then on and so on our last match we ran again not perfect because of the code we had running but we ran the whole match with few lost packets. The only thing I can say is that you shouldn't connect the DS to the robot prior to going onto the field with out turning the robot off afterward. We will check our radios and crio at the school when I get a chance. Trying to find the original culprit. BTW this is only our second year and are just starting to learn about the drivers log etc.
vBulletin® v3.6.4, Copyright ©2000-2017, Jelsoft Enterprises Ltd.