Chief Delphi

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)
-   C/C++ (http://www.chiefdelphi.com/forums/forumdisplay.php?f=183)
-   -   Intermitten power failures (http://www.chiefdelphi.com/forums/showthread.php?t=87516)

RandomIteration 14-11-2010 16:57
Intermitten power failures
 
Hello I am from Team 2941, and over a rather recent time frame, I have converted our robot from Labview to C++. For the most part, the robot has worked solidly (with that being said, I have stored previous versions of the robot that do not suffer such sporadic power failures).

Well to the point now outside of the background information; between some time interval (~5 seconds over wireless; really random on wired, ten seconds to a whole minute) the cRio cuts power to all the systems so the victors and solenoids will flash for a moment (no power is being sent through). I was originally programming with pthreads (avoid watchdog use where I can :D ), and have ultimately removed all threads except for the robot drive. At the current state, it is

GetWatchdog().SetEnabled(False);
while( TeleOperated()){
Wait(0.05);
}

The diagnostic light blinks every 200ms, for about 200ms; however, we do not have the jumper hooked up acrossed it. (Mind you, the only thread running is now the Teleoperated main). We have flashed the cRio with Windriver, swapped out our Distribution board, sidecar, and victors for jaguars.

So with that monster of a story, anyone have thoughts or answers?

Robototes2412 14-11-2010 17:43
Re: Intermitten power failures
 
I would try disabling every part on the robot physically and slowly reenable them until you find the bad component, its usually a rouge jaguar or something

Joe Ross 14-11-2010 17:47
Re: Intermitten power failures
 
You can disable the user watchdog, but you can't disable the system (communications) watchdog. Is there anything printed on the diagnostics tab of the driver station? Does it do it with the default code?

Robototes2412 14-11-2010 17:57
Re: Intermitten power failures
 
i mean unplug everything non-essential and plug stuff in till it breaks

Radical Pi 14-11-2010 23:46
Re: Intermitten power failures
 
are you attempting to run the camera? We experienced a similar fault during the season because of our frame pushing on the camera reset button. If there are any references to the camera class at all in your code and the camera is not connected or otherwise unable to communicate, the camera task will consume enough resources to trip the system watchdog.

Greg McKaskle 15-11-2010 07:52
Re: Intermitten power failures
 
Just a minor clarification. I believe the camera classes will timeout when the camera is down or not connected. If you have the camera code in the same thread that feeds the watchdog, your watchdog will be fed late, but the CPU load will not be high. If the camera code is in its own thread/task, the camera code will slowly return errors, with no impact on the rest of the robot.

Greg McKaskle

Al Skierkiewicz 15-11-2010 09:02
Re: Intermitten power failures
 
Greg,
Can you check with NI Rep Andrew on this one? I believe we found that under certain conditions, loops with in user programs caused a slight lag in program execution which caused a watchdog issue. This was a common problem throughout the season last year regardless of which language was used. Due to the error, all output was halted for a brief period. It manifests as a power loss but is not. Conditions included loss of robot indicator, loss of motor drive, compressor shuts off, robot twitches, etc.

RandomIteration 15-11-2010 11:06
Re: Intermitten power failures
 
Sorry little more to add to this; we have swapped out the power distribution board, the sidecar, jaguars, taken the camera code out, and probably something else I have forgetten.

This has produced nearly the same results (but if it is a communication error, and say it were the communication watchdog, is there way around it?)

Ether 15-11-2010 12:41
Re: Intermitten power failures
 
Quote:
Originally Posted by RandomIteration (Post 980789)
Hello I am from Team 2941, and over a rather recent time frame, I have converted our robot from Labview to C++. For the most part, the robot has worked solidly (with that being said, I have stored previous versions of the robot that do not suffer such sporadic power failures).
Just curious, and perhaps my reading of what you wrote above is not what you intended, but have you tried re-loading one of these "stored previous versions of the robot that does not suffer such sporadic power failures" and see if it still works?

Quote:
while( TeleOperated()){
Wait(0.05);
}
A bit off-topic, but why are you running Teleop at 20Hz ?

Radical Pi 15-11-2010 12:52
Re: Intermitten power failures
 
Quote:
Originally Posted by Greg McKaskle (Post 980842)
Just a minor clarification. I believe the camera classes will timeout when the camera is down or not connected. If you have the camera code in the same thread that feeds the watchdog, your watchdog will be fed late, but the CPU load will not be high. If the camera code is in its own thread/task, the camera code will slowly return errors, with no impact on the rest of the robot.

Greg McKaskle
This did not appear to be the case on our robot. The only reference to the camera was in the (C++) constructor so that camera images would be forwarded to our robot. Watching the output on NetConsole, errors were flooding and appeared to have no timeout. Another symptom we encountered with this was the clock on the Driver Station freezing with the robot but no watchdog error was being shown in the bottom left corner. This was also running with a Serial-based CAN.

RandomIteration 15-11-2010 22:01
Re: Intermitten power failures
 
Quote:
Originally Posted by Ether (Post 980879)
Just curious, and perhaps my reading of what you wrote above is not what you intended, but have you tried re-loading one of these "stored previous versions of the robot that does not suffer such sporadic power failures" and see if it still works?
And yeah, I must have poorly typed that. We went back to older code (simpler ones) to see if it was software.

Quote:
A bit off-topic, but why are you running Teleop at 20Hz ?
Honestly, toned it down cause I wasn't sure if the usual 50hz was stressing the cRio. Well guess not (part of the whole debugging thing :) ) .

RandomIteration 15-11-2010 22:02
Re: Intermitten power failures
 
Oh and mind you, I had removed all traces of the camera (as ours is broken; circuit boad gets stressed too much, causes camera to fail).

Greg McKaskle 16-11-2010 08:27
Re: Intermitten power failures
 
Camera communication is over TCP, and both the reads and writes have timeouts of 0.5 or 1 second on LV. I can't say what they are on C++. This was my basis for saying that I didn't expect a large CPU spike with the camera wasn't plugged in or powered.

When I said slow errors, that is of course not very definitive. It is true that last year's system could choke a bit when a large stream of fast errors were being sent to the DS. I don't think those errors would be only camera, but if you had lots of other errors, the camera could contribute to make things even worse.

As for loops causing watchdog issues, it isn't the fact that it is a loop, but the time that the code takes to execute. If the code manipulates a solenoid, a common occurrence last year, they code would often have delays and would cause a watchdog each time it ran. Even worse, it would cause many teleop packets to be overwritten since the teleop handler would take far longer than 20ms to execute.

A printout was inserted deep into the libraries last year to help determine when short disables were occurring. It kept count of the user and system watchdog disables since boot. If these correspond with the robot "power failures", then indeed, the WD was disabling the robot due to a missed deadline. If not, then it was something else.

Greg McKaskle


All times are GMT -5. The time now is 12:30.

Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi