FIRST is really looking into the Einstein problems

[JB987]

Originally Posted by Al Skierkiewicz

Joe,
The CPU likely was not the problem but anything is possible. How did you interface to the control system?

I agree with you, Al. Not likely that our on board cpu contributed to the "connections" issue but was just letting you know there was one robot on Einstein for sure that was using cpu. It processed Kinect data and sent key data to Crio over router. The Crio then sent info to driver station via the bridge. As I recall, programmers had compacted data so it would be sending reduced data packets and avoid over loading transmissions.

[trilogy2826]

One additional data point to add to the pile that I don't believe was mentioned in any of this years previous "Communincations Issues" posts:

Specs:
Bridge Firmware: 1.21
DS Laptop: HP Elite
Axis camera
Vision targeting through separate PC connected through ethernet into bridge
C++
4 port cRio

When 2826 arrived on Archimedes for the first practice match:
- We noticed an extremely high, relatively constant trip time (> 150ms) through the diagnostic tab on the DS.
- Our FMS connection indicator would rapidly turn off and on and our axis camera feed was extremely lagged.
- For the first 30 seconds, we were pretty much dead in the water.
- We "jiggled" the ethernet connection and check CPU and network utilization, and the numbers were low and consistent with times where no errors ocurred
- We then unplugged and re-plugged our ethernet connection to the DS laptop mid match. For an unknown reason, everything suddenly worked perfectly. No latency, no lag and full control
-Afterward, the FTA mentioned that our trip time suddenly decreased per their diagnostics, so that was consistent

For Practice Match 2:
- We had the same rapid disconnection issues as match 1 as soon as we connected the ethernet cable to the DS laptop
- The FTA came to us before the match was started and we tried unplugging and re-plugging the ethernet cable. This did not fix the issue, same symptoms
- The FTA's curiousity was piqued, so he allowed a bit of troubleshooting
- We could ping the bot with consistently high latency
- We disabled and enabled the DS laptop network adapter - No fix
- The FMS was rebooted - No Fix
- Robot was rebooted - No Fix
-DS laptop OS (windows 7, obviously) was restarted - At this point we had a completely new behavior in that we could ping the robot with low latency, but the FMS would not show connection. Quote from FTA "What the heck?! I've never seen this before"
-We started the match anyway and had no control or feedback from/to the bot
-About 10 seconds into the match, we disconnected and reconnected the ethernet cable and viola! Everything worked again.

*At this time, we had a consistent set of failures and symptoms, but no consistent correction method. We were starting to freak out...

For Practice Match 3:
- Immediately prior to approaching the Alliance station, we rebooted the DS laptop
-We waited for at least 1 minute for the robot and bridge to boot
-We waited for the DS and dashboard applications to fully launch.
-We confirmed that "2826" was established on the LCD board above our alliance station
-We plugged in the ethernet cable into the DS laptop and got immediate connection to the robot and FMS. Little to no latency (<2-3 ms on average)
-we ran the match with no glitches or issues

At this point we knew our DS laptop was somehow perpetuating the issue, if not directly causing it. We ran the process described in 3 more practice matches, all Quals and elims with absolutely no issues.

We know that we did not solve anything, but hopefully this process can shed some light on other noted issues.

[slijin]

Originally Posted by trilogy2826

At this point we knew our DS laptop was somehow perpetuating the issue, if not directly causing it. We ran the process described in 3 more practice matches, all Quals and elims with absolutely no issues.

On a similar point, we've also seen the DS software itself to be faulty, where only by closing and reopening the DS could a connection to either the robot or Cypress board be reestablished.

[Thad House]

Originally Posted by JB987

I agree with you, Al. Not likely that our on board cpu contributed to the "connections" issue but was just letting you know there was one robot on Einstein for sure that was using cpu. It processed Kinect data and sent key data to Crio over router. The Crio then sent info to driver station via the bridge. As I recall, programmers had compacted data so it would be sending reduced data packets and avoid over loading transmissions.

I also have heard the 118 uses a BeagleBone to process their camera.

[DMetalKong]

I saw this posted in another thread. I have not seen this specific issue mentioned in this thread so far, but it seemed relevant so I thought I would bring it up.

Originally Posted by taichichuan

We used an Arduino Mega with an Ethernet shield this year for the RPM on the shooter and the angle of our ball launch. Using the Ethernet wasn't too hard. But, the cRio had something odd in the robot code where if you started sending packets, but there was nothing actively reading them, that the robot would eventually go offline. We were using UDP on the local link as a directed unicast to keep packets from going across the link to the driver station.

It's definitely related to the robot code because I've been using VxWorks for over 25 years and I've never encountered anything like this failure in the main OS. Nonetheless, if you have a thread reading the packets all of the time, then everything works well. I'll be doing some tests as soon as I get access to a cRio again to see how much data the cRio can absorb via the Ethernet during normal operation before the driver station packets start getting dropped.

[linuxboy]

Originally Posted by JB987

I agree with you, Al. Not likely that our on board cpu contributed to the "connections" issue but was just letting you know there was one robot on Einstein for sure that was using cpu. It processed Kinect data and sent key data to Crio over router. The Crio then sent info to driver station via the bridge. As I recall, programmers had compacted data so it would be sending reduced data packets and avoid over loading transmissions.

When you say that the computer processing the Kinect data sent key data to the cRIO over the router, I'm assuming you mean the switch on the bridge, right? The only real device with routing tables in the standard control system is the field access point.

[JB987]

Originally Posted by linuxboy

When you say that the computer processing the Kinect data sent key data to the cRIO over the router, I'm assuming you mean the switch on the bridge, right? The only real device with routing tables in the standard control system is the field access point.

Yes.

[Hjelstrom]

Originally Posted by linuxboy

When you say that the computer processing the Kinect data sent key data to the cRIO over the router, I'm assuming you mean the switch on the bridge, right? The only real device with routing tables in the standard control system is the field access point.

Ok, forgive me if I describe anything incorrectly, this was our team's first experiment with writing our own network communications code (among many other things this year). We have a Pandaboard on the robot and it has an ethernet cable connecting it to the bridge. We use TCP/IP packets and the Pandaboard acts as a listening server and the cRio as a client (C++). Packets are only sent from the Pandaboard to the cRio when the cRio sends a packet requesting information. There should be no extraneous packets being sent and its all over the wire, not wireless. We've used it this way in every match all year at two regionals and at worlds. The only problem we had was one match in Las Vegas where our Pandaboard did not power up and the cRio waits up to 30s trying to establish a connection. We know the behavior of that problem and that is not what happened on Einstein. We also had one match in Curie where we did not plug our robot battery all the way in. So we had no "unknown" problems even in practice matches all year until Einstein. I don't know if any of us ever looked at what firmware our bridge is running.

I think what Joe was referring to was that we also went to extra effort this year to minimize the data we were sending from the cRio to the driver station. We set up a toggle button on the driver station controls that enables and disables data sending (disabled by default). The reason for this is that we noticed that the driver station quickly becomes very "laggy" when we send data to it (this is using SmartDashboard and NetworkTables). First we tried to solve this by upgrading the laptop but it still happened. So we worked around the problem by not sending data unless we really needed to "debug". Even with our data sending enabled, we were using a timer to drastically limit how often we send data. It seems very odd to me that the small amount of data being sent from the robot to the DS could make any difference in the DS performance.

We stuck with SmartDashboard because for another part of our tele-op control system, the flexibility of being able to send a small amount of data from the SmartDashboard to the robot was so useful (i.e. the functionality of NetworkTables). It was worth all of the trouble we had to go through and all of the workarounds like restarting our DS before each match.

[JB987]

Originally Posted by Hjelstrom

Ok, forgive me if I describe anything incorrectly, this was our team's first experiment with writing our own network communications code (among many other things this year). We have a Pandaboard on the robot and it has an ethernet cable connecting it to the bridge. We use TCP/IP packets and the Pandaboard acts as a listening server and the cRio as a client (C++). Packets are only sent from the Pandaboard to the cRio when the cRio sends a packet requesting information. There should be no extraneous packets being sent and its all over the wire, not wireless. We've used it this way in every match all year at two regionals and at worlds. The only problem we had was one match in Las Vegas where our Pandaboard did not power up and the cRio waits up to 30s trying to establish a connection. We know the behavior of that problem and that is not what happened on Einstein. We also had one match in Curie where we did not plug our robot battery all the way in. So we had no "unknown" problems even in practice matches all year until Einstein. I don't know if any of us ever looked at what firmware our bridge is running.

I think what Joe was referring to was that we also went to extra effort this year to minimize the data we were sending from the cRio to the driver station. We set up a toggle button on the driver station controls that enables and disables data sending (disabled by default). The reason for this is that we noticed that the driver station quickly becomes very "laggy" when we send data to it (this is using SmartDashboard and NetworkTables). First we tried to solve this by upgrading the laptop but it still happened. So we worked around the problem by not sending data unless we really needed to "debug". Even with our data sending enabled, we were using a timer to drastically limit how often we send data. It seems very odd to me that the small amount of data being sent from the robot to the DS could make any difference in the DS performance.

We stuck with SmartDashboard because for another part of our tele-op control system, the flexibility of being able to send a small amount of data from the SmartDashboard to the robot was so useful (i.e. the functionality of NetworkTables). It was worth all of the trouble we had to go through and all of the workarounds like restarting our DS before each match.

Thanks Greg...so much better than my second hand translation

[BHOP]

Thank you FIRST!

I think they should also figure out a way to get (or simulate) a couple thousand (10s of thousands?) WiFi receiving/transmitting devices like cell phones, laptops, etc into their test.

It seemed there were more problems as more people started watching matches, in the elims and then accumulating to einstein where everyone was surroundinng one field with their wireless devices.

or maybe FIRST should just license some white space spectrum

[Al Skierkiewicz]

There were actually very few on 2.4 GHz. Maybe 4 or 5. The field runs on 5 GHz and there are very few phones that can transmit that at this time.

[Libby K]

A great big step in the right direction.

Thank you, FIRST.

[DonRotolo]

Originally Posted by Al Skierkiewicz

The only thing that can't be duplicated is the raging storm outside in St. Louis and the power distribution in the dome.

If FIRST was really committed, they'd fly the dome in, and use one of Dean's weather-controlling machines.

Originally Posted by BHOP

Thank you FIRST!

I think they should also figure out a way to get (or simulate) a couple thousand (10s of thousands?) WiFi receiving/transmitting devices like cell phones, laptops, etc into their test.


or maybe FIRST should just license some white space spectrum

RF interference can easily be simulated. Ten thousand devices look like a high level of RF noise to any other device, so just broadcasting some wideband RF will effectively duplicate this condition. This is a common practice in the industry.

Maybe instead FIRST should require every team member and volunteer to get their Amateur Radio license, and we can then just use the Ham bands

In all seriousness: FIRST is tossing significant resources at the issue, and I for one am very comfortable with the process they are following so far. Kudos for good communications, too.

[EricS-Team180]

Originally Posted by DonRotolo

Maybe instead FIRST should require every team member and volunteer to get their Amateur Radio license,

.
Funny you should say that, Don. Bryce, who's attending for S.P.A.M. just so happens to have his Ham license

FRC 180 is totally behind these efforts by F.I.R.S.T , and we sure hope that the testing will yield insight into what I'll just call "the comms problem". It has been vexing us for awhile, now.

[Andy Brockway]

.
-DS laptop OS (windows 7, obviously)

We are still running the 2009 classmate with XP. I have installed every update. None of the thumb drives from FIRST have worked to reflash the drive. The first year FIRST sent me a new one but still I could not reflash.

My team has had no connectivity issues. We are using the older bridge and firmware.