Week one regional control system showstoppers

[sparrowkc]

We just finished day two of the Kansas City regional. Although we had some bright spots today (We're currently tied for the highest match score of this season ), two problems continualy plagued our robot's performance and left us dead in the water during at least three matches. I'm hoping to hear from teams that have experienced the same problems and what they are doing to solve them.

First Problem
On several occasions, we experienced severe lag (~5 seconds) between the driver controls and the robot's motion. This problem only emerges when we're on the field with the FMS. Sometimes it persists after a match across multiple power cycles, and then fixes itself seemingly randomly. I'm fairly confident that this is nota code problem; We've never seen it before and is solved by multiple reboots, not code changes.

Second problem
During some matches, our joystick simply did not work. Inputs on the IO board did work, and the joystick was listed in the setup tab of the DS. I know of at least one other team that had this problem. I remember it happening on and off the field. It seemed to persist across power cycles and then fix itself. Ours was a logitech 3-axis joystick. It was going through the hub, and I couldn't reproduce it after I connected it straight to the classmate.

So... Ideas? Comments?

[nighterfighter]

1: During a practice match at Peachtree on Thursday, the FMS locked us out.
Turns out the people running the FMS had assigned us to be on both red AND blue, so our drive station was freaking out, requiring a reboot of both robot and driver station.

2: Our E-Stop seems to randomly connect. Our hub provided seemed to have stopped functioning about a week before competition, so now we use a different hub, but on occasion the Classmate does not detect it.

3: Incredibly long boot-times.

4: After each match, you have to use the red "Exit" button on the DS, or else it will stay in the "FMS Locked" mode.

5:And of course, no power supplies on the field for the Classmate.
In one of today's matches, there was an hour and a half delay, (Apparently the FMS crashed or something), leaving teams waiting in line to go for an hour and a half, with thier DS's ready to go.

[JBotAlan]

I'm sure FIRST knows of our woes. We are all irritated by the same plague of problems, but I imagine that the best we can do is suck it up.

I've seen some oddness when sleeping the DS, so I would not suggest doing that. However, figuring that it is about a 2-minute cold boot time, it should be okay to turn off the DS until the match before you is over. Then, right when the buzzer goes off, turn on the DS. In most cases, you should be alright.

As for the radio slowness, the best I've seen is to power on the 'bot and leave it alone. I did see a lot of teams stationary for the entire match with a fast-flashing RSL. That would not make me happy.

Which leads me to another point. Has anyone else noticed a bug in the RSL code? About 40 percent of the 'bots I saw (anecdotally, not empirically) would be acting just fine, but fast-flashing the light. Isn't it supposed to be "long on, flicker off, long on, flicker off" during teleop?

Overall, considering how complex the system is, we are lucky it works as well as it does. I question some of the decisions made, but many engineers (who have been engineering since before I was born) have poked at the intricacies of this system, I'm sure.

I am also sure things are going to get a lot better. This morning, we had a delay of several minutes between matches. Towards the end of the day, however, things were running fairly smoothly. Stuff happens every year. There's a lot to work out.

Now, did you have fun?

[Chris is me]

Originally Posted by sparrowkc

First Problem
On several occasions, we experienced severe lag (~5 seconds) between the driver controls and the robot's motion. This problem only emerges when we're on the field with the FMS. Sometimes it persists after a match across multiple power cycles, and then fixes itself seemingly randomly. I'm fairly confident that this is nota code problem; We've never seen it before and is solved by multiple reboots, not code changes.

This issue happened to 1714 at IRI last year. I don't know how to describe what caused it, but I believe data was being written to memory that was not erased before power off or something like that. I guess there was data involving the camera that ate up memory because the camera was no longer plugged in. If you don't know how to change your code to fix this (if I were more code savvy I'd have a better description here), flashing your cRIO every match will make the problem go away as a quick fix.

[Mike Betts]

Originally Posted by JBotAlan

...Which leads me to another point. Has anyone else noticed a bug in the RSL code? About 40 percent of the 'bots I saw (anecdotally, not empirically) would be acting just fine, but fast-flashing the light. Isn't it supposed to be "long on, flicker off, long on, flicker off" during teleop?...

I noted the same phenomena at our local pre-ship scrimmage. I brought it to FIRST's attention but I would guess it's fairly low on their radar screen right now.

[apalrd]

Lots of field problems at Kettering. We notices the same GIANT delay on the field, blamed it on the camera, removed the camera, and were fine. We have had several instances where BOOT TIMES and CODE DOWNLOAD TIMES have caused us giant problems, but those existed last year. Our biggest issue by far is that if you power on the Classmate with the FMS and Cypress board connected, it will go into FMS Lock before detecting the Cypress board and not detect the Cypress board at all. The fix is even worse. You must reboot your Classmate two times without FMS connection to get the IO board back. We are no longer shutting down our laptop ever again. We played two matches out of 7 (and won them both, btw) with only our driver, since our operators box was completely dead. What is also annoying is there is no way to detect this loss, as the Classmate sends the last good data packet to the cRio even if the Cypress board is unplugged, so we can't pull an input high and wait for it to go low. We had so many FMS problems today that matches went 1/2 hour late, and they're still around 10 matches behind.

[Vikesrock]

Originally Posted by apalrd

What is also annoying is there is no way to detect this loss, as the Classmate sends the last good data packet to the cRio even if the Cypress board is unplugged, so we can't pull an input high and wait for it to go low. We had so many FMS problems today that matches went 1/2 hour late, and they're still around 10 matches behind.

There are ways, just probably not ways that are as simple as you'd like/as simple as they should be.

If you have an input that changes once for each DS packet period you should be able to write code that will properly detect a missing PSoC.

This doesn't change the fact that there should be some type of easy to check flag set by the Driver's Station when the PSoC is not prsent, I am merely pointing out that it would be possible to do the detection yourself.

[Mark McLeod]

These weren't the typical always rapid flashing you get when the RSL is miswired without the LA-to-Lb jumper?

Originally Posted by JBotAlan

Which leads me to another point. Has anyone else noticed a bug in the RSL code? About 40 percent of the 'bots I saw (anecdotally, not empirically) would be acting just fine, but fast-flashing the light. Isn't it supposed to be "long on, flicker off, long on, flicker off" during teleop?

Originally Posted by Mike Betts

I noted the same phenomena at our local pre-ship scrimmage. I brought it to FIRST's attention but I would guess it's fairly low on their radar screen right now.

[billbo911]

Originally Posted by sparrowkc

We just finished day two of the Kansas City regional. Although we had some bright spots today (We're currently tied for the highest match score of this season ), two problems continualy plagued our robot's performance and left us dead in the water during at least three matches. I'm hoping to hear from teams that have experienced the same problems and what they are doing to solve them.

First Problem
On several occasions, we experienced severe lag (~5 seconds) between the driver controls and the robot's motion. This problem only emerges when we're on the field with the FMS. Sometimes it persists after a match across multiple power cycles, and then fixes itself seemingly randomly. I'm fairly confident that this is nota code problem; We've never seen it before and is solved by multiple reboots, not code changes.

Second problem
During some matches, our joystick simply did not work. Inputs on the IO board did work, and the joystick was listed in the setup tab of the DS. I know of at least one other team that had this problem. I remember it happening on and off the field. It seemed to persist across power cycles and then fix itself. Ours was a logitech 3-axis joystick. It was going through the hub, and I couldn't reproduce it after I connected it straight to the classmate.

So... Ideas? Comments?

This sounds very similar to the issues we had until we replaced our USB hub. For less than $10 you might be able to make these problems go away!!

[Alan Anderson]

Originally Posted by Mark McLeod

These weren't the typical always rapid flashing you get when the RSL is miswired without the LA-to-Lb jumper?

Perhaps not. I was helping a team Thursday, and their Digital Sidecar's RSL was blinking fast while their robot was enabled and running fine. I don't remember the specifics, but I think their cRIO was imaged with v19 and they had the latest Driver Station update. After making everything up to date, the RSL was normal.

The amber RSL jumper was my first thought here, though.

[Joe Ross]

Originally Posted by Alan Anderson

Perhaps not. I was helping a team Thursday, and their Digital Sidecar's RSL was blinking fast while their robot was enabled and running fine. I don't remember the specifics, but I think their cRIO was imaged with v19 and they had the latest Driver Station update. After making everything up to date, the RSL was normal.

I believe there is another RSL flash pattern (possibly undocumented) when it's attached to the FMS that indicates that there is a SW version mismatch. It may be a fast flash, I don't remember. Someone found it during the beta when they tried to use this year's controller with last year's FMS.

[Mike Betts]

Originally Posted by Mark McLeod

These weren't the typical always rapid flashing you get when the RSL is miswired without the LA-to-Lb jumper?

Mark,

No they were not. I personally inspected several robots, including the two I mentor, and the jumper was installed. Moreover, the RSL acted correctly in the pits, both before and after the team was on the field, and the RSL only misbehaved under FMS control.

Regards,

Mike

[lynca]

Originally Posted by apalrd

Lots of field problems at Kettering.

We also had huge delays and communication problems at Bayou even during elimination matches. I believe some of this pain comes from the new Driver Station bugs. Make sure all teams upgrade to the latest driver station update !

With the old version everything will appear to be working with comm & code lights but nothing will work. This happened many times on Thursday.

[DonRotolo]

See also this thread, at NJ a few teams found that double-connecting the USB Hub might have fixed that kind of problem.

[whytheheckme]

Originally Posted by Alan Anderson

I don't remember the specifics, but I think their cRIO was imaged with v19 and they had the latest Driver Station update. After making everything up to date, the RSL was normal.

I was sitting at the Scorekeeper table for part of the day at the CT Scrimmage, where we observed the 'out-of-place' fast-blinking RSL, and I'm pretty sure I remember checking the version that the DS was reporting as v20 (at least in one case that I remember checking). Not to say that there aren't multiple things that could be causing this.

Also, with regard to the First Problem, what type of compression/frame size are you using? It could be a bandwidth issue on the field, or just simply a different throughput of field equipment vs your kit router? (simply un-based thoughts, btw)

Jacob