Week one regional control system showstoppers

[sparrowkc]

We just finished day two of the Kansas City regional. Although we had some bright spots today (We're currently tied for the highest match score of this season ), two problems continualy plagued our robot's performance and left us dead in the water during at least three matches. I'm hoping to hear from teams that have experienced the same problems and what they are doing to solve them.

First Problem
On several occasions, we experienced severe lag (~5 seconds) between the driver controls and the robot's motion. This problem only emerges when we're on the field with the FMS. Sometimes it persists after a match across multiple power cycles, and then fixes itself seemingly randomly. I'm fairly confident that this is nota code problem; We've never seen it before and is solved by multiple reboots, not code changes.

Second problem
During some matches, our joystick simply did not work. Inputs on the IO board did work, and the joystick was listed in the setup tab of the DS. I know of at least one other team that had this problem. I remember it happening on and off the field. It seemed to persist across power cycles and then fix itself. Ours was a logitech 3-axis joystick. It was going through the hub, and I couldn't reproduce it after I connected it straight to the classmate.

So... Ideas? Comments?

[nighterfighter]

1: During a practice match at Peachtree on Thursday, the FMS locked us out.
Turns out the people running the FMS had assigned us to be on both red AND blue, so our drive station was freaking out, requiring a reboot of both robot and driver station.

2: Our E-Stop seems to randomly connect. Our hub provided seemed to have stopped functioning about a week before competition, so now we use a different hub, but on occasion the Classmate does not detect it.

3: Incredibly long boot-times.

4: After each match, you have to use the red "Exit" button on the DS, or else it will stay in the "FMS Locked" mode.

5:And of course, no power supplies on the field for the Classmate.
In one of today's matches, there was an hour and a half delay, (Apparently the FMS crashed or something), leaving teams waiting in line to go for an hour and a half, with thier DS's ready to go.

[JBotAlan]

I'm sure FIRST knows of our woes. We are all irritated by the same plague of problems, but I imagine that the best we can do is suck it up.

I've seen some oddness when sleeping the DS, so I would not suggest doing that. However, figuring that it is about a 2-minute cold boot time, it should be okay to turn off the DS until the match before you is over. Then, right when the buzzer goes off, turn on the DS. In most cases, you should be alright.

As for the radio slowness, the best I've seen is to power on the 'bot and leave it alone. I did see a lot of teams stationary for the entire match with a fast-flashing RSL. That would not make me happy.

Which leads me to another point. Has anyone else noticed a bug in the RSL code? About 40 percent of the 'bots I saw (anecdotally, not empirically) would be acting just fine, but fast-flashing the light. Isn't it supposed to be "long on, flicker off, long on, flicker off" during teleop?

Overall, considering how complex the system is, we are lucky it works as well as it does. I question some of the decisions made, but many engineers (who have been engineering since before I was born) have poked at the intricacies of this system, I'm sure.

I am also sure things are going to get a lot better. This morning, we had a delay of several minutes between matches. Towards the end of the day, however, things were running fairly smoothly. Stuff happens every year. There's a lot to work out.

Now, did you have fun?

[Mike Betts]

Originally Posted by JBotAlan

...Which leads me to another point. Has anyone else noticed a bug in the RSL code? About 40 percent of the 'bots I saw (anecdotally, not empirically) would be acting just fine, but fast-flashing the light. Isn't it supposed to be "long on, flicker off, long on, flicker off" during teleop?...

I noted the same phenomena at our local pre-ship scrimmage. I brought it to FIRST's attention but I would guess it's fairly low on their radar screen right now.

[apalrd]

Lots of field problems at Kettering. We notices the same GIANT delay on the field, blamed it on the camera, removed the camera, and were fine. We have had several instances where BOOT TIMES and CODE DOWNLOAD TIMES have caused us giant problems, but those existed last year. Our biggest issue by far is that if you power on the Classmate with the FMS and Cypress board connected, it will go into FMS Lock before detecting the Cypress board and not detect the Cypress board at all. The fix is even worse. You must reboot your Classmate two times without FMS connection to get the IO board back. We are no longer shutting down our laptop ever again. We played two matches out of 7 (and won them both, btw) with only our driver, since our operators box was completely dead. What is also annoying is there is no way to detect this loss, as the Classmate sends the last good data packet to the cRio even if the Cypress board is unplugged, so we can't pull an input high and wait for it to go low. We had so many FMS problems today that matches went 1/2 hour late, and they're still around 10 matches behind.

[Vikesrock]

Originally Posted by apalrd

What is also annoying is there is no way to detect this loss, as the Classmate sends the last good data packet to the cRio even if the Cypress board is unplugged, so we can't pull an input high and wait for it to go low. We had so many FMS problems today that matches went 1/2 hour late, and they're still around 10 matches behind.

There are ways, just probably not ways that are as simple as you'd like/as simple as they should be.

If you have an input that changes once for each DS packet period you should be able to write code that will properly detect a missing PSoC.

This doesn't change the fact that there should be some type of easy to check flag set by the Driver's Station when the PSoC is not prsent, I am merely pointing out that it would be possible to do the detection yourself.

[lynca]

Originally Posted by apalrd

Lots of field problems at Kettering.

We also had huge delays and communication problems at Bayou even during elimination matches. I believe some of this pain comes from the new Driver Station bugs. Make sure all teams upgrade to the latest driver station update !

With the old version everything will appear to be working with comm & code lights but nothing will work. This happened many times on Thursday.

[DtD]

Originally Posted by apalrd

Lots of field problems at Kettering. We notices the same GIANT delay on the field, blamed it on the camera, removed the camera, and were fine. We have had several instances where BOOT TIMES and CODE DOWNLOAD TIMES have caused us giant problems, but those existed last year. Our biggest issue by far is that if you power on the Classmate with the FMS and Cypress board connected, it will go into FMS Lock before detecting the Cypress board and not detect the Cypress board at all. The fix is even worse. You must reboot your Classmate two times without FMS connection to get the IO board back. We are no longer shutting down our laptop ever again. We played two matches out of 7 (and won them both, btw) with only our driver, since our operators box was completely dead. What is also annoying is there is no way to detect this loss, as the Classmate sends the last good data packet to the cRio even if the Cypress board is unplugged, so we can't pull an input high and wait for it to go low. We had so many FMS problems today that matches went 1/2 hour late, and they're still around 10 matches behind.

^^This. Our team (Which uses C++) could never get the camera to work, we also found out after our first match that the camera randomly decided to either cause huge robot reaction delays or constant robot resetting (Which causes even larger delays - about 5 seconds)

As for joysticks and the io board not working, it might be a power draw issue. We never had the issue but we are not using the io board.

~David

[Mark McLeod]

These weren't the typical always rapid flashing you get when the RSL is miswired without the LA-to-Lb jumper?

Originally Posted by JBotAlan

Which leads me to another point. Has anyone else noticed a bug in the RSL code? About 40 percent of the 'bots I saw (anecdotally, not empirically) would be acting just fine, but fast-flashing the light. Isn't it supposed to be "long on, flicker off, long on, flicker off" during teleop?

Originally Posted by Mike Betts

I noted the same phenomena at our local pre-ship scrimmage. I brought it to FIRST's attention but I would guess it's fairly low on their radar screen right now.

[Alan Anderson]

Originally Posted by Mark McLeod

These weren't the typical always rapid flashing you get when the RSL is miswired without the LA-to-Lb jumper?

Perhaps not. I was helping a team Thursday, and their Digital Sidecar's RSL was blinking fast while their robot was enabled and running fine. I don't remember the specifics, but I think their cRIO was imaged with v19 and they had the latest Driver Station update. After making everything up to date, the RSL was normal.

The amber RSL jumper was my first thought here, though.

[Joe Ross]

Originally Posted by Alan Anderson

Perhaps not. I was helping a team Thursday, and their Digital Sidecar's RSL was blinking fast while their robot was enabled and running fine. I don't remember the specifics, but I think their cRIO was imaged with v19 and they had the latest Driver Station update. After making everything up to date, the RSL was normal.

I believe there is another RSL flash pattern (possibly undocumented) when it's attached to the FMS that indicates that there is a SW version mismatch. It may be a fast flash, I don't remember. Someone found it during the beta when they tried to use this year's controller with last year's FMS.

[heydowns]

Originally Posted by Joe Ross

I believe there is another RSL flash pattern (possibly undocumented) when it's attached to the FMS that indicates that there is a SW version mismatch. It may be a fast flash, I don't remember. Someone found it during the beta when they tried to use this year's controller with last year's FMS.

That was me. The blink code for cRIO image version mismatch is unfortunately largely indistinguishable from "no comms". The "mismatch" here is between what the actual cRIO image version is and what FMS wants it to be.
You shouldn't see any of this after practice day, as this is an inspection item.

[Radical Pi]

For the first problem, during my testing of our bot I found the lag in control to be caused by an enormous number of printf's in WPILib code under certain error circumstances (camera code looking for non-existent camera, throwing an error on every loop of the camera task, CAN bus not communicating with a jag that should be there for some reason such as breaker out or ID reset). I've written my share of lengthy complaint posts about it.

One time we lost communications with the cypress board right before a match that was hugely delayed, causing the computer to sleep and (as far as I could tell) crash the service for the cypress. A simple reboot fixed everything.

I've found the cypress board to have caused a variety of issues this year, one of which caused our 2 alliance partners to completely fail in eliminations. With all of these young, energetic programmers around, you'd think FIRST could create a more resilient system

[whytheheckme]

Originally Posted by Alan Anderson

I don't remember the specifics, but I think their cRIO was imaged with v19 and they had the latest Driver Station update. After making everything up to date, the RSL was normal.

I was sitting at the Scorekeeper table for part of the day at the CT Scrimmage, where we observed the 'out-of-place' fast-blinking RSL, and I'm pretty sure I remember checking the version that the DS was reporting as v20 (at least in one case that I remember checking). Not to say that there aren't multiple things that could be causing this.

Also, with regard to the First Problem, what type of compression/frame size are you using? It could be a bandwidth issue on the field, or just simply a different throughput of field equipment vs your kit router? (simply un-based thoughts, btw)

Jacob

[sparrowkc]

Originally Posted by whytheheckme

...Also, with regard to the First Problem, what type of compression/frame size are you using? It could be a bandwidth issue on the field, or just simply a different throughput of field equipment vs your kit router? (simply un-based thoughts, btw)

Jacob

320x240 compression=30 fps=10

We used these settings because they were a reasonable margin below the settings that caused >1sec lag on the classmate. I doubt bandwidth was the problem. Chrisisme's answer makes the most sense to me right now, we need to look into that some more. We ended up scrapping all of our camera code and physically unplugging the camera. The lag never came back, but who knows if it was the camera's fault or not...

On day two of the KC regional, we switched to a different usb hub and never saw the joystick problem again. We had been using the kit hub plugged into both ports-- the new one only plugs into one port so we pluged the joystick straight into the other port. At our next regional we probably won't be using a hub at all since we were told how to run without the USB Estop (Double click the stop button indicator light in the diagnostics tab of the DS software).