Week one regional control system showstoppers

We just finished day two of the Kansas City regional. Although we had some bright spots today (We’re currently tied for the highest match score of this season :slight_smile: ), two problems continualy plagued our robot’s performance and left us dead in the water during at least three matches. I’m hoping to hear from teams that have experienced the same problems and what they are doing to solve them.

First Problem
On several occasions, we experienced severe lag (~5 seconds) between the driver controls and the robot’s motion. This problem only emerges when we’re on the field with the FMS. Sometimes it persists after a match across multiple power cycles, and then fixes itself seemingly randomly. I’m fairly confident that this is nota code problem; We’ve never seen it before and is solved by multiple reboots, not code changes.

Second problem
During some matches, our joystick simply did not work. Inputs on the IO board did work, and the joystick was listed in the setup tab of the DS. I know of at least one other team that had this problem. I remember it happening on and off the field. It seemed to persist across power cycles and then fix itself. Ours was a logitech 3-axis joystick. It was going through the hub, and I couldn’t reproduce it after I connected it straight to the classmate.

So… Ideas? Comments?

1: During a practice match at Peachtree on Thursday, the FMS locked us out.
Turns out the people running the FMS had assigned us to be on both red AND blue, so our drive station was freaking out, requiring a reboot of both robot and driver station.

2: Our E-Stop seems to randomly connect. Our hub provided seemed to have stopped functioning about a week before competition, so now we use a different hub, but on occasion the Classmate does not detect it.

3: Incredibly long boot-times.

4: After each match, you have to use the red “Exit” button on the DS, or else it will stay in the “FMS Locked” mode.

5:**And of course, no power supplies on the field for the Classmate. **
In one of today’s matches, there was an hour and a half delay, :ahh: (Apparently the FMS crashed or something), leaving teams waiting in line to go for an hour and a half, with thier DS’s ready to go.

I’m sure FIRST knows of our woes. We are all irritated by the same plague of problems, but I imagine that the best we can do is suck it up.

I’ve seen some oddness when sleeping the DS, so I would not suggest doing that. However, figuring that it is about a 2-minute cold boot time, it should be okay to turn off the DS until the match before you is over. Then, right when the buzzer goes off, turn on the DS. In most cases, you should be alright.

As for the radio slowness, the best I’ve seen is to power on the 'bot and leave it alone. I did see a lot of teams stationary for the entire match with a fast-flashing RSL. That would not make me happy.

Which leads me to another point. Has anyone else noticed a bug in the RSL code? About 40 percent of the 'bots I saw (anecdotally, not empirically) would be acting just fine, but fast-flashing the light. Isn’t it supposed to be “long on, flicker off, long on, flicker off” during teleop?

Overall, considering how complex the system is, we are lucky it works as well as it does. I question some of the decisions made, but many engineers (who have been engineering since before I was born) have poked at the intricacies of this system, I’m sure.

I am also sure things are going to get a lot better. This morning, we had a delay of several minutes between matches. Towards the end of the day, however, things were running fairly smoothly. Stuff happens every year. There’s a lot to work out.

Now, did you have fun?

This issue happened to 1714 at IRI last year. I don’t know how to describe what caused it, but I believe data was being written to memory that was not erased before power off or something like that. I guess there was data involving the camera that ate up memory because the camera was no longer plugged in. If you don’t know how to change your code to fix this (if I were more code savvy I’d have a better description here), flashing your cRIO every match will make the problem go away as a quick fix.

I noted the same phenomena at our local pre-ship scrimmage. I brought it to FIRST’s attention but I would guess it’s fairly low on their radar screen right now.

Lots of field problems at Kettering. We notices the same GIANT delay on the field, blamed it on the camera, removed the camera, and were fine. We have had several instances where BOOT TIMES and CODE DOWNLOAD TIMES have caused us giant problems, but those existed last year. Our biggest issue by far is that if you power on the Classmate with the FMS and Cypress board connected, it will go into FMS Lock before detecting the Cypress board and not detect the Cypress board at all. The fix is even worse. You must reboot your Classmate two times without FMS connection to get the IO board back. We are no longer shutting down our laptop ever again. We played two matches out of 7 (and won them both, btw) with only our driver, since our operators box was completely dead. What is also annoying is there is no way to detect this loss, as the Classmate sends the last good data packet to the cRio even if the Cypress board is unplugged, so we can’t pull an input high and wait for it to go low. We had so many FMS problems today that matches went 1/2 hour late, and they’re still around 10 matches behind.

There are ways, just probably not ways that are as simple as you’d like/as simple as they should be.

If you have an input that changes once for each DS packet period you should be able to write code that will properly detect a missing PSoC.

This doesn’t change the fact that there should be some type of easy to check flag set by the Driver’s Station when the PSoC is not prsent, I am merely pointing out that it would be possible to do the detection yourself.

These weren’t the typical always rapid flashing you get when the RSL is miswired without the LA-to-Lb jumper?

This sounds very similar to the issues we had until we replaced our USB hub. For less than $10 you might be able to make these problems go away!!

Perhaps not. I was helping a team Thursday, and their Digital Sidecar’s RSL was blinking fast while their robot was enabled and running fine. I don’t remember the specifics, but I think their cRIO was imaged with v19 and they had the latest Driver Station update. After making everything up to date, the RSL was normal.

The amber RSL jumper was my first thought here, though.

I believe there is another RSL flash pattern (possibly undocumented) when it’s attached to the FMS that indicates that there is a SW version mismatch. It may be a fast flash, I don’t remember. Someone found it during the beta when they tried to use this year’s controller with last year’s FMS.

Mark,

No they were not. I personally inspected several robots, including the two I mentor, and the jumper was installed. Moreover, the RSL acted correctly in the pits, both before and after the team was on the field, and the RSL only misbehaved under FMS control.

Regards,

Mike

We also had huge delays and communication problems at Bayou even during elimination matches. I believe some of this pain comes from the new Driver Station bugs. Make sure all teams upgrade to the latest driver station update !

With the old version everything will appear to be working with comm & code lights but nothing will work. This happened many times on Thursday.

See also this thread, at NJ a few teams found that double-connecting the USB Hub might have fixed that kind of problem.

I was sitting at the Scorekeeper table for part of the day at the CT Scrimmage, where we observed the ‘out-of-place’ fast-blinking RSL, and I’m pretty sure I remember checking the version that the DS was reporting as v20 (at least in one case that I remember checking). Not to say that there aren’t multiple things that could be causing this.

Also, with regard to the First Problem, what type of compression/frame size are you using? It could be a bandwidth issue on the field, or just simply a different throughput of field equipment vs your kit router? (simply un-based thoughts, btw)

Jacob

^^This. Our team (Which uses C++) could never get the camera to work, we also found out after our first match that the camera randomly decided to either cause huge robot reaction delays or constant robot resetting (Which causes even larger delays - about 5 seconds)

As for joysticks and the io board not working, it might be a power draw issue. We never had the issue but we are not using the io board.

~David

320x240 compression=30 fps=10

We used these settings because they were a reasonable margin below the settings that caused >1sec lag on the classmate. I doubt bandwidth was the problem. Chrisisme’s answer makes the most sense to me right now, we need to look into that some more. We ended up scrapping all of our camera code and physically unplugging the camera. The lag never came back, but who knows if it was the camera’s fault or not…

On day two of the KC regional, we switched to a different usb hub and never saw the joystick problem again. We had been using the kit hub plugged into both ports-- the new one only plugs into one port so we pluged the joystick straight into the other port. At our next regional we probably won’t be using a hub at all since we were told how to run without the USB Estop (Double click the stop button indicator light in the diagnostics tab of the DS software).

You should try 320x240 with a compression of 40. Shouldn’t be a significant difference in quality, and that will use significantly less bandwidth. Just a thought though, no basis for this other than some simple experimentation. Do you also have the updated dashboard code without the graphs etc. that were really slowing down with the camera running?

Jacob

We also had major communication problems in KC this weekend. This FMS / Control system is NOT ready for prime time. I posted a rather lengthy post in another thread about this. For me I am frustrated to no end over the overall poor quality of the complete control system. Too many bugs and tweaks with no real answer from one owner.

We had issues on Thursday connecting to FMS. Issues so bad that they loaned us another classmate (which promptly messed up our controls … not sure why).

We fixed our connection issues by moving our radio, uninstalling the driver station, reinstalling the driver station (from a different source than our own), and reinstalling the updates (again a different source than our origional).

No issues after that.