FIRST is really looking into the Einstein problems

[SillyRabbitt]

I am a mentor/coach for a rookie team and I just have one question about the final 12 headed to FIRST. How many of the robots in question that lost communication during the matches had on-board cpu's? Its just a thought that might be an issue. One more x-file!

[Al Skierkiewicz]

Originally Posted by SillyRabbitt

I am a mentor/coach for a rookie team and I just have one question about the final 12 headed to FIRST. How many of the robots in question that lost communication during the matches had on-board cpu's? Its just a thought that might be an issue. One more x-file!

Sorry I didn't answer this before...
In reality "losing communications" is a catch all for what problems actually took place. We only know the robots stopped moving. We know (knew at the time) some of the robots that stopped moving were connected to the field and the RSL lights on the robots showed they were enabled. The problem(s) is(are) far more complex than you might think.
Very few robots this year had on board computers in addition to the Crio. I inspected none in four events. Those teams that really wanted to incorporate the Kinect on the robot used them and a few others for vision processing. I don't remember seeing any on Einstein but it was pretty dark in the pit.

[Taylor]

Originally Posted by Al Skierkiewicz

Sorry I didn't answer this before...
In reality "losing communications" is a catch all for what problems actually took place. We only know the robots stopped moving. We know (knew at the time) some of the robots that stopped moving were connected to the field and the RSL lights on the robots showed they were enabled. The problem(s) is(are) far more complex than you might think.
Very few robots this year had on board computers in addition to the Crio. I inspected none in four events. Those teams that really wanted to incorporate the Kinect on the robot used them and a few others for vision processing. I don't remember seeing any on Einstein but it was pretty dark in the pit.

Our "losing communications" moments at QCR involved 3 matches of a working hybrid mode, then as soon as teleop started, no comms, no code. One match, about 3 seconds of hybrid mode, then no comms, no code. One match - as soon as the match began, no comms, no code. Four of the five matches, a connection was reestablished with about :10 left in the match; the last one, we never had connectivity during the match. These issues appeared at different stations and different alliances. When we reset the router firmware from v1.4 to v1.2 it appeared to fix the problems (the last two matches, we saw no communication issues). We had no comm issues at BMR, during practice at QCR, or the first half of Friday at QCR. We had no extra onboard computer.
As Al said, the "losing communications" catchall really envelops a lot of symptoms and issues, and ours -seems to- have been traced back to an individual robot issue.
It is a tangled web indeed, and I do not envy those charged with unraveling it. As has been said before, I hope a bumpercrop of identifiable and repeatable issues pop up in June.

Sorry to hijack the Einstein thread with a regional vignette, but I felt it was part of the greater picture.

[Al Skierkiewicz]

Originally Posted by Taylor

When we reset the router firmware from v1.4 to v1.2 it appeared to fix the problems (the last two matches, we saw no communication issues). We had no comm issues at BMR, during practice at QCR, or the first half of Friday at QCR.

Anybody else experience this? Taylor is it possible that the firmware was corrupt? Had you tried re-flashing the router with the v1.4? Not to hijack the thread, but all of you can PM me with this info. Thanks.

[Holtzman]

Originally Posted by Al Skierkiewicz

Anybody else experience this? Taylor is it possible that the firmware was corrupt? Had you tried re-flashing the router with the v1.4? Not to hijack the thread, but all of you can PM me with this info. Thanks.

At GTR-W, all the teams with firmware 1.4 had difficulties connecting to the field whenever there were more than 3 robots on the field at a time. We were one of about a half a dozen teams that had to downgrade our router firmware to 1.21. The problem was first noticed on Thursday afternoon when we started getting more than 3 robots at a time for practice matches. Anytime a robot with Firmware 1.4 went out on the field and powered on, ping times for all teams would spike, and robots would start randomly dropping out and reconnecting. The running theory was that the combination of a large number of access points, and the something in the 1.4 firmware was causing communication to drop in and out. By late Thursday afternoon, everybody downgraded to 1.21, and the event ran smoothly.

Figuring out that it was specific robots that was causing the problem was tricky. We couldn't figure out how something specific to our robot was causing so many difficulties for all the other teams on the field, especially since we had already attended two other regionals, and run without a single issue at either. The FTA asked all the teams that seemed to be causing the field difficulties to check their firmware, and we realized it was the one thing we all had in common. Thinking back, I can't recall if it was something that inspectors checked that at the Championship. Does anyone else remember being asked router firmware at the Championship?

It was one of the things that each Einstein team was asked in the questionnaire we filled out.

[Chris Fultz]

We had issues at SMR with 1.4 and stopped in 3 matches.

We made other changes also to reduce the CPU demand, but also downgraded to 1.21 and did not have problems at BMR or CHP.

[JohnBoucher]

This was noted back in March
http://www.chiefdelphi.com/forums/sh...1&postcount=35

[Alan Anderson]

Sometime after Week 4 of regional competitons, CSAs were asked to check the WPA kiosk logs to identify teams with D-Link firmware version 1.40 and to watch for issues. Maybe not all of them got the memo.

[steelerborn]

Just a thought, but I couldn't help but notice that there was a lot of spare time between when the divisions finished playing and the start of Einstein. It would be a good idea to have 6 bots from the divisions, that did not make it to Einstein, play a few rounds just to test the field systems? You could have a raffle to see which teams make it to the pre-Einstein "just for fun" event, this way the people in the stands have something to watch (although the paper airplanes were a good distraction ), and also if the field does have any issues you would be able to catch them before the first matches start. And you can just use leftover basketballs (or some other game piece for next year) from the divisions, so Einstein has a fresh field still.

[Tom Bottiglieri]

At CMP, we ran laggy controls (robot responded 2 seconds after operator input, tons of dropped packets, long trip times) on 1.4. Downgraded to 1.21 and all was well.

[EricVanWyk]

Originally Posted by class1234567

I dont know if im saying this right but when every my team (1985) and 1986 where on the same field at the same time the FMS whould shut down and this happened at GKC at STL regionals so i was wondering is FIRST looking into this as well

Originally Posted by Tom Bottiglieri

At CMP, we ran laggy controls (robot responded 2 seconds after operator input, tons of dropped packets, long trip times) on 1.4. Downgraded to 1.21 and all was well.

These should also be reported to the email address in Jon's letter.

[trilogy2826]

One additional data point to add to the pile that I don't believe was mentioned in any of this years previous "Communincations Issues" posts:

Specs:
Bridge Firmware: 1.21
DS Laptop: HP Elite
Axis camera
Vision targeting through separate PC connected through ethernet into bridge
C++
4 port cRio

When 2826 arrived on Archimedes for the first practice match:
- We noticed an extremely high, relatively constant trip time (> 150ms) through the diagnostic tab on the DS.
- Our FMS connection indicator would rapidly turn off and on and our axis camera feed was extremely lagged.
- For the first 30 seconds, we were pretty much dead in the water.
- We "jiggled" the ethernet connection and check CPU and network utilization, and the numbers were low and consistent with times where no errors ocurred
- We then unplugged and re-plugged our ethernet connection to the DS laptop mid match. For an unknown reason, everything suddenly worked perfectly. No latency, no lag and full control
-Afterward, the FTA mentioned that our trip time suddenly decreased per their diagnostics, so that was consistent

For Practice Match 2:
- We had the same rapid disconnection issues as match 1 as soon as we connected the ethernet cable to the DS laptop
- The FTA came to us before the match was started and we tried unplugging and re-plugging the ethernet cable. This did not fix the issue, same symptoms
- The FTA's curiousity was piqued, so he allowed a bit of troubleshooting
- We could ping the bot with consistently high latency
- We disabled and enabled the DS laptop network adapter - No fix
- The FMS was rebooted - No Fix
- Robot was rebooted - No Fix
-DS laptop OS (windows 7, obviously) was restarted - At this point we had a completely new behavior in that we could ping the robot with low latency, but the FMS would not show connection. Quote from FTA "What the heck?! I've never seen this before"
-We started the match anyway and had no control or feedback from/to the bot
-About 10 seconds into the match, we disconnected and reconnected the ethernet cable and viola! Everything worked again.

*At this time, we had a consistent set of failures and symptoms, but no consistent correction method. We were starting to freak out...

For Practice Match 3:
- Immediately prior to approaching the Alliance station, we rebooted the DS laptop
-We waited for at least 1 minute for the robot and bridge to boot
-We waited for the DS and dashboard applications to fully launch.
-We confirmed that "2826" was established on the LCD board above our alliance station
-We plugged in the ethernet cable into the DS laptop and got immediate connection to the robot and FMS. Little to no latency (<2-3 ms on average)
-we ran the match with no glitches or issues

At this point we knew our DS laptop was somehow perpetuating the issue, if not directly causing it. We ran the process described in 3 more practice matches, all Quals and elims with absolutely no issues.

We know that we did not solve anything, but hopefully this process can shed some light on other noted issues.

[Robogineer1649]

Originally Posted by steelerborn

Just a thought, but I couldn't help but notice that there was a lot of spare time between when the divisions finished playing and the start of Einstein. It would be a good idea to have 6 bots from the divisions, that did not make it to Einstein, play a few rounds just to test the field systems? You could have a raffle to see which teams make it to the pre-Einstein "just for fun" event, this way the people in the stands have something to watch (although the paper airplanes were a good distraction ), and also if the field does have any issues you would be able to catch them before the first matches start. And you can just use leftover basketballs (or some other game piece for next year) from the divisions, so Einstein has a fresh field still.

I like your idea but potentially one of the robots could damage the field not just the game pieces, but i noticed that the bridges tended to become damaged from certain bridge manipulators. Wheels also tended to scrape up the plastic covering that the bridges had on them.

[JB987]

Originally Posted by Al Skierkiewicz

Sorry I didn't answer this before...
In reality "losing communications" is a catch all for what problems actually took place. We only know the robots stopped moving. We know (knew at the time) some of the robots that stopped moving were connected to the field and the RSL lights on the robots showed they were enabled. The problem(s) is(are) far more complex than you might think.
Very few robots this year had on board computers in addition to the Crio. I inspected none in four events. Those teams that really wanted to incorporate the Kinect on the robot used them and a few others for vision processing. I don't remember seeing any on Einstein but it was pretty dark in the pit.

Al,
Our robot has an on board cpu (Panda Board) for our on board Kinect and we never had any problems until Einstein.

[Al Skierkiewicz]

Joe,
The CPU likely was not the problem but anything is possible. How did you interface to the control system?