Week one regional control system showstoppers
 

We just finished day two of the Kansas City regional. Although we had some bright spots today (We're currently tied for the highest match score of this season :) ), two problems continualy plagued our robot's performance and left us dead in the water during at least three matches. I'm hoping to hear from teams that have experienced the same problems and what they are doing to solve them.

First Problem
On several occasions, we experienced severe lag (~5 seconds) between the driver controls and the robot's motion. This problem only emerges when we're on the field with the FMS. Sometimes it persists after a match across multiple power cycles, and then fixes itself seemingly randomly. I'm fairly confident that this is nota code problem; We've never seen it before and is solved by multiple reboots, not code changes.

Second problem
During some matches, our joystick simply did not work. Inputs on the IO board did work, and the joystick was listed in the setup tab of the DS. I know of at least one other team that had this problem. I remember it happening on and off the field. It seemed to persist across power cycles and then fix itself. Ours was a logitech 3-axis joystick. It was going through the hub, and I couldn't reproduce it after I connected it straight to the classmate.

So... Ideas? Comments?

Re: Week one regional control system showstoppers
 

1: During a practice match at Peachtree on Thursday, the FMS locked us out.
Turns out the people running the FMS had assigned us to be on both red AND blue, so our drive station was freaking out, requiring a reboot of both robot and driver station.

2: Our E-Stop seems to randomly connect. Our hub provided seemed to have stopped functioning about a week before competition, so now we use a different hub, but on occasion the Classmate does not detect it.

3: Incredibly long boot-times.

4: After each match, you have to use the red "Exit" button on the DS, or else it will stay in the "FMS Locked" mode.

5:And of course, no power supplies on the field for the Classmate.
In one of today's matches, there was an hour and a half delay, :ahh: (Apparently the FMS crashed or something), leaving teams waiting in line to go for an hour and a half, with thier DS's ready to go.

Re: Week one regional control system showstoppers
 

I'm sure FIRST knows of our woes. We are all irritated by the same plague of problems, but I imagine that the best we can do is suck it up.

I've seen some oddness when sleeping the DS, so I would not suggest doing that. However, figuring that it is about a 2-minute cold boot time, it should be okay to turn off the DS until the match before you is over. Then, right when the buzzer goes off, turn on the DS. In most cases, you should be alright.

As for the radio slowness, the best I've seen is to power on the 'bot and leave it alone. I did see a lot of teams stationary for the entire match with a fast-flashing RSL. That would not make me happy.

Which leads me to another point. Has anyone else noticed a bug in the RSL code? About 40 percent of the 'bots I saw (anecdotally, not empirically) would be acting just fine, but fast-flashing the light. Isn't it supposed to be "long on, flicker off, long on, flicker off" during teleop?

Overall, considering how complex the system is, we are lucky it works as well as it does. I question some of the decisions made, but many engineers (who have been engineering since before I was born) have poked at the intricacies of this system, I'm sure.

I am also sure things are going to get a lot better. This morning, we had a delay of several minutes between matches. Towards the end of the day, however, things were running fairly smoothly. Stuff happens every year. There's a lot to work out.

Now, did you have fun?

Re: Week one regional control system showstoppers
 

This issue happened to 1714 at IRI last year. I don't know how to describe what caused it, but I believe data was being written to memory that was not erased before power off or something like that. I guess there was data involving the camera that ate up memory because the camera was no longer plugged in. If you don't know how to change your code to fix this (if I were more code savvy I'd have a better description here), flashing your cRIO every match will make the problem go away as a quick fix.

Re: Week one regional control system showstoppers
 

I noted the same phenomena at our local pre-ship scrimmage. I brought it to FIRST's attention but I would guess it's fairly low on their radar screen right now.

Re: Week one regional control system showstoppers
 

Lots of field problems at Kettering. We notices the same GIANT delay on the field, blamed it on the camera, removed the camera, and were fine. We have had several instances where BOOT TIMES and CODE DOWNLOAD TIMES have caused us giant problems, but those existed last year. Our biggest issue by far is that if you power on the Classmate with the FMS and Cypress board connected, it will go into FMS Lock before detecting the Cypress board and not detect the Cypress board at all. The fix is even worse. You must reboot your Classmate two times without FMS connection to get the IO board back. We are no longer shutting down our laptop ever again. We played two matches out of 7 (and won them both, btw) with only our driver, since our operators box was completely dead. What is also annoying is there is no way to detect this loss, as the Classmate sends the last good data packet to the cRio even if the Cypress board is unplugged, so we can't pull an input high and wait for it to go low. We had so many FMS problems today that matches went 1/2 hour late, and they're still around 10 matches behind.

Re: Week one regional control system showstoppers
 

There are ways, just probably not ways that are as simple as you'd like/as simple as they should be.

If you have an input that changes once for each DS packet period you should be able to write code that will properly detect a missing PSoC.

This doesn't change the fact that there should be some type of easy to check flag set by the Driver's Station when the PSoC is not prsent, I am merely pointing out that it would be possible to do the detection yourself.

Re: Week one regional control system showstoppers
 

These weren't the typical always rapid flashing you get when the RSL is miswired without the LA-to-Lb jumper?

Re: Week one regional control system showstoppers
 

This sounds very similar to the issues we had until we replaced our USB hub. For less than $10 you might be able to make these problems go away!!

Re: Week one regional control system showstoppers
 

Perhaps not. I was helping a team Thursday, and their Digital Sidecar's RSL was blinking fast while their robot was enabled and running fine. I don't remember the specifics, but I think their cRIO was imaged with v19 and they had the latest Driver Station update. After making everything up to date, the RSL was normal.

The amber RSL jumper was my first thought here, though.

Re: Week one regional control system showstoppers
 

I believe there is another RSL flash pattern (possibly undocumented) when it's attached to the FMS that indicates that there is a SW version mismatch. It may be a fast flash, I don't remember. Someone found it during the beta when they tried to use this year's controller with last year's FMS.

Re: Week one regional control system showstoppers
 

Mark,

No they were not. I personally inspected several robots, including the two I mentor, and the jumper was installed. Moreover, the RSL acted correctly in the pits, both before and after the team was on the field, and the RSL only misbehaved under FMS control.

Regards,

Mike

Re: Week one regional control system showstoppers
 

We also had huge delays and communication problems at Bayou even during elimination matches. I believe some of this pain comes from the new Driver Station bugs. Make sure all teams upgrade to the latest driver station update !

With the old version everything will appear to be working with comm & code lights but nothing will work. This happened many times on Thursday.

Re: Week one regional control system showstoppers
 

See also this thread, at NJ a few teams found that double-connecting the USB Hub might have fixed that kind of problem.

Re: Week one regional control system showstoppers
 

I was sitting at the Scorekeeper table for part of the day at the CT Scrimmage, where we observed the 'out-of-place' fast-blinking RSL, and I'm pretty sure I remember checking the version that the DS was reporting as v20 (at least in one case that I remember checking). Not to say that there aren't multiple things that could be causing this.

Also, with regard to the First Problem, what type of compression/frame size are you using? It could be a bandwidth issue on the field, or just simply a different throughput of field equipment vs your kit router? (simply un-based thoughts, btw)

Jacob

Re: Week one regional control system showstoppers
 

^^This. Our team (Which uses C++) could never get the camera to work, we also found out after our first match that the camera randomly decided to either cause huge robot reaction delays or constant robot resetting (Which causes even larger delays - about 5 seconds)

As for joysticks and the io board not working, it might be a power draw issue. We never had the issue but we are not using the io board.

~David

Re: Week one regional control system showstoppers
 

320x240 compression=30 fps=10

We used these settings because they were a reasonable margin below the settings that caused >1sec lag on the classmate. I doubt bandwidth was the problem. Chrisisme's answer makes the most sense to me right now, we need to look into that some more. We ended up scrapping all of our camera code and physically unplugging the camera. The lag never came back, but who knows if it was the camera's fault or not...

On day two of the KC regional, we switched to a different usb hub and never saw the joystick problem again. We had been using the kit hub plugged into both ports-- the new one only plugs into one port so we pluged the joystick straight into the other port. At our next regional we probably won't be using a hub at all since we were told how to run without the USB Estop (Double click the stop button indicator light in the diagnostics tab of the DS software).

Re: Week one regional control system showstoppers
 

You should try 320x240 with a compression of 40. Shouldn't be a significant difference in quality, and that will use significantly less bandwidth. Just a thought though, no basis for this other than some simple experimentation. Do you also have the updated dashboard code without the graphs etc. that were really slowing down with the camera running?

Jacob

Re: Week one regional control system showstoppers
 

We also had major communication problems in KC this weekend. This FMS / Control system is NOT ready for prime time. I posted a rather lengthy post in another thread about this. For me I am frustrated to no end over the overall poor quality of the complete control system. Too many bugs and tweaks with no real answer from one owner.

Re: Week one regional control system showstoppers
 

We had issues on Thursday connecting to FMS. Issues so bad that they loaned us another classmate (which promptly messed up our controls ... not sure why).

We fixed our connection issues by moving our radio, uninstalling the driver station, reinstalling the driver station (from a different source than our own), and reinstalling the updates (again a different source than our origional).

No issues after that.

Re: Week one regional control system showstoppers
 

As I mentioned in the Kansas City thread, we learned that our problem during at least one match was an IP address conflict with another team. This was evidently confirmed after the fact by reviewing field data. I would refer everyone to our post in the thread link below regarding the proper timing and sequence for logging into your DS and powering up your robot on the field.

http://www.chiefdelphi.com/forums/sh...ad.php?t=83813

Re: Week one regional control system showstoppers
 

We had our stop button die on use during the competition. It had happened once or twice during testing, but went away after a reboot so we figured it was a code issue.

After some probing with the multimeter, it turns out that we had a busted connector on the end of the stop button USB cable. The ground connector was losing contact intermittently. A quick run to Wal-Mart Friday night (snuck past the manager 5 minutes after closing :D ), and some soldering Saturday morning, and we had spliced in a new connector on. That solved the problem for the weekend.

If you open up the stop button, they are very cheaply made. There is a piece of kite string in the cable that serves to give it tensile strength. Our had solder balls and burned bits on the board. I think I'll open it up tonight and clean it out now that I'm home and have a good soldering station.

Re: Week one regional control system showstoppers
 

For all those having trouble with the KoP stop button, it is possible to run in the pits without it. On the driver station navigate to the tab called diagnostics, and a list of items plugged in should appear with red and green gots beside them. double click on the (presumably) red dot next to the "Stop Button" label and choose the option "Yes, I'll be careful". A dialog box will then coundown 20 seconds and then the robot should be enableable without a stop button plugged in. We had our button die on us as well in the pits after travel, but due to errors in packing had not the solder materials needed to fix it :) we'll learn for LA regional I guess.

Re: Week one regional control system showstoppers
 

We saw the Classmate pegged at 100% cpu usage running the dashboard. We offloaded the dashboard to another PC and found out that FIRST did not provide a second Ethernet cable for a Dashboard PC. They do not allow a hub/switch to be connected between the classmate and the field. We were told that we needed to use a USB-To-Ethernet adapter (slow), we bridged the network cards together and it worked fine in the pits and on the field but the FMS refused to let us connect to it. So we had to remove the network bridge and do without for the weekend. :mad: Whats really sad is the field switch still has a couple empty Ethernet ports and a cable would be so simple.

Re: Week one regional control system showstoppers
 

In the pre-season, we've figured out an annoying, yet seemingly foolproof way to fix the joystick thing in a pinch.

If you go into the DS, under the setup tab, you can see all the joysticks that the computer detects. If you reorder the malfunctioning one to a different position, and then move it back, this seems to fix the problem.

Ultimately I wish we knew a better fix, but this has worked in the several times we've experienced this problem.

And our Targus hub went out, too. We're planning to bring a few spares;)

Re: Week one regional control system showstoppers
 

That was me. The blink code for cRIO image version mismatch is unfortunately largely indistinguishable from "no comms". The "mismatch" here is between what the actual cRIO image version is and what FMS wants it to be.
You shouldn't see any of this after practice day, as this is an inspection item.

Re: Week one regional control system showstoppers
 

For the first problem, during my testing of our bot I found the lag in control to be caused by an enormous number of printf's in WPILib code under certain error circumstances (camera code looking for non-existent camera, throwing an error on every loop of the camera task, CAN bus not communicating with a jag that should be there for some reason such as breaker out or ID reset). I've written my share of lengthy complaint posts about it.

One time we lost communications with the cypress board right before a match that was hugely delayed, causing the computer to sleep and (as far as I could tell) crash the service for the cypress. A simple reboot fixed everything.

I've found the cypress board to have caused a variety of issues this year, one of which caused our 2 alliance partners to completely fail in eliminations. With all of these young, energetic programmers around, you'd think FIRST could create a more resilient system