Problems with cRIO and driving while competing

Technical Control System
1

Hello,

This weekend, our FIRST team had their event, and it could have gone better.

We are using labview, and with our programming, we were getting poor driving performance and loss of communication. We received help from many people at the event, and still were not able to load or shoot the ball.

To troubleshoot, we swapped out the radio, cRIO, digital sidecar, switched to a different laptop from the classmate, changed network cables, disconnected the digital sidecar, ran tethered in the pits, all to no avail.:ahh:

We then reimaged the cRIO, and loaded the base code for an arcade drive robot with no edits at all.

We still have lost packets that can precede latency issues, latency issues followed by packet loss, all of which can happen while the robot is not being controlled but enabled and tethered. This is the watchdog error that shows:

ERROR <Code< -44061 occurred at “Left and Right Motors” in the VI path: Robot Main.vi FRC: The loop that contains RobotDrive is not running fast enough. This error can occur if the loop contains too much code, or if one or more other loops are starving the RobotDrive loop.

During the event, we were able to drive, but not very reliably. We were told by someone to press F1 to bring the control back faster, and that seemed to work a little.

Our team is very disappointed and would like to be more competitive in our next event in 2 weeks.

Any help would be appreciated, and more info will be provided as needed.

2

My team encouters these errors every match. The only thing that helps reduce these errors is to have most of our code in periodic tasks, with one fast loop(10ms) and one slow loop (100ms) for our various tasks.

3

What is the Driver Station CPU usage? It is displayed on the charts tab of the driver station (or you can open Task Manager). It is not in the DS log, however.

I helped one team this week who’s netbook had unexpectedly high CPU usage. They were running around 50% usage when idle. Adding the DS and dashboard brought the CPU usage to 100%. Closing the dashboard got it to 70% and their lost packets and latency decreased dramatically.

Can you post a representative Driver Station log? See http://wpilib.screenstepslive.com/s/3120/m/8559/l/97119-driver-station-log-file-viewer

4

Please post your log files or an image of the log file. I’m curious to see your CPU usage and other errors. To Joe’s point, what laptop was used for the DS. Did you happen to notice its CPU usage?

As I’ve said many times, reimaging is something I rarely recommend. If the controller doesn’t boot, reimage it. Otherwise, the time is probably better spent debugging and experimenting with other element.

Greg McKaskle

5

We were using a Classmate that came with the KOP, as well as a Dell XPS laptop.

Here is the base logs and our original code, and the base code for just a driving base.

Original code is the 2014 Robot Project while catapult.zip
Base code is the 2014 Robot Project drive only.zip

logs from competition.zip (608 KB)
2014 Robot Project drive only.zip (2.93 MB)
2014 Robot Project while catapult.zip (208 KB)


logs from competition.zip (608 KB)
2014 Robot Project drive only.zip (2.93 MB)
2014 Robot Project while catapult.zip (208 KB)

6

The Thursday logs show a frequent stream of errors coming from teleop trying to read a Digital Input refnum that doesn’t exist. This caused the cRIO CPU to be pegged near 100%.

On Friday, that was resolved, and the errors were just timeouts on RobotDrive during auto.

The 5:30 log on Friday has a one second disabled gap during teleop, but it isn’t clear what caused that. The comms were clean and CPU was low.

The three qualification matches on Saturday look clean.

Saturday afternoon around 2:00, there were some tests in the pits where the auto timeouts started happening again

I haven’t had a chance to look at the code yet, but the logs do not show communications drops, cRIO CPU usage or errors to be the cause.

Greg McKaskle

7

Looking at the code, the first project looks exactly like the template code. Did you have issues driving with the default code?

The other project has one point that looks funny to me. The catapult motor is being controlled in two parallel loops. If the logic is consistent, then this shouldn’t be a problem, but it clearly allows for funky interactions if the logic or drivers are doing things the programmers didn’t expect.

I searched to see if anyone else is updating the X or Y globals. I didn’t see any in the code that was posted. So I don’t think this is causing any issues. Another way to do this is simply to read the joystick in the periodic loop.

So at this point, I don’t see anything obvious. Can you point out periods in the log file or with specific mechanisms that were problematic? Perhaps I’m not looking at the correct logs?

Greg McKaskle

8

The problem we were having while driving is lack of control of the robot, especially driving not responding to command.

Here is one log when we switched to another laptop. The packet loss is what we were told was causing the lack of response for steering.

2014_03_21 18_10_01 Fri.zip (5.69 KB)


2014_03_21 18_10_01 Fri.zip (5.69 KB)

9

That is very little packet loss.

You only attached the .dslog file and not the .dsevent file, so I can’t look at errors, but your packet loss never tops 5 lost packets over 1/2 second intervals.

I’ve seen better, but I’ve seen much worse. I see no evidence that your robot dropped communications. I see no evidence that your robot outputs were shutdown because of packet loss.

Greg McKaskle

10

Here is both of the logs for our replacement laptop. The original classmate logs are attached above. If the logs are not showing any issues, why is it that the robot would not respond to the controls for seconds at time, making us a sitting duck on the field?

We ended up just driving and trying to defend, where the robot had been designed to shoot as well as pass.

Log Files.zip (6.44 KB)


Log Files.zip (6.44 KB)

11

As Joe Ross said, the logs don’t record the laptop’s CPU load. If it’s bogged down trying to do too much, there’s a good chance the Driver Station will get behind on its communication tasks. If that happens, you’ll get control delays, with the lag time very likely increasing as the match goes on.

Are you using a custom Dashboard?

12

Default dashboard. With most of the work being done on the cRIO, how much is the laptop actually doing? It is the default dashboard, shouldn’t the laptop be able to do this without issue? If it cannot, why are we getting inferior equipment?

13

You aren’t trying to log on to both the Developer and Driver accounts simultaneously, are you?

14

No, only driver, and log out, not quick switch when changing between.

Sorry if the previous post sounded nasty, just really frustrated. Thanks for all the suggestions.

15

I looked at the latest zip. You have one error related to reading the color enabled on the camera at boot up time. Other than that, it is a clean log.

From the battery voltage, I can see that your robot didn’t move during auto, and there are a good number of two and three second flat spots where your robot was sitting still.

As Joe mentioned, I’d look at the laptop CPU usage and I’d also look to verify that the ribbon cable was firmly seated at both ends. I’d make sure that the power to the digital breakout board wasn’t going in and out or that I didn’t have a short to it. I’d also make sure the digital module is fully inserted and the clips are holding it in place.

Greg McKaskle

16

Auto never did work on the field with our programming. The only time auto worked was in the pits with the robot on blocks.

The flat spots are the problem we are having, the driver is trying to get the robot to move, and there is no response from the robot.

The camera was removed about halfway thru our matches, so that may be that error, but the unresponsiveness was there throughout.

All of the cables were checked and were seated, as were the modules. We tested the sidecar with a multimeter.

17

We have comm and response problems, if anti-virus is enabled.
Just another thing to check.

18

Looking at your logs, it appears that packets are arriving at the DS when they are supposed to. It is of course to good to look at CPU usage and keep background services like virus scanning in check, but I believe your problem is electrical.

There is a new feature of the dashboard that I’ve seen hardly anyone use. It lets you record a match, video, network tables, joysticks, motor outputs, even Kinect and stuff. If you had a recorded match, it would let you see what your joystick values were during your flat spot. It would let you see what your SW was asking the motor controllers to output. But I’m sure you don’t.

So. To read the tea leaves again. I’d do two things to your robot. Follow the wiring from one of your motor controllers back to the PD. Verify that it is plugged into a 40A circuit. It isn’t common, but sometimes the motors are running on a 20A or 30A circuit. They work when driven nicely, and they trip breakers when driven like you mean it. Then they sit still while the motor controller reboots – usually a second or two depending on the model and mfg.

The second thing I’d check, though it doesn’t explain the flat spots as well. Verify that your digital breakout has three bright green LEDs next to the 12V input power, and of course make sure the 12V power is actually connected to a 12V circuit.

Greg McKaskle

19

To everyone who has watched us troubleshoot this issue, thank you for your suggestions.

We were able to get some test time while on a field, and it seems we were blowing the breakers on our drive train. Driving in the pits or around practice, the drivers were being careful of obstacles and not driving all-out like during matches.

Greg,
Where is the miracle software in the dashboard that sees all and tells all?