The 8.2 (or 8.3) Battery Voltage Bug

[eugenebrooks]

We are using a combination of Kevin Watson's camera code, using only the sensor reporting, not the servo functionality, and a timer/sensor interrupt to regulate the speed of our ball shooting wheel. We see the camera go off line sporadically, with some of the printf() statements ceasing to appear on the upload line. When this happens, the battery voltage shown on the OI is 8.2 (or 8.3) even though the voltage of the main battery is just fine.

We are backing out of the camera code, which depends on interrupts for the serial line support and our own interrupt based feedback for the ball wheel speed, but are interested in knowing if any other teams have seen this problem. It started off being very sporadic, but for some minor code adjustments can be something that hits almost 100% of the time.

We have integrated the new library code that fixes an interrupt problem
but this did not fix this bug for us.

Eugene

[Mark McLeod]

We had the same symptoms before our robot went in the box. Our other interrupt lines were detached, only running one timer interrupt and it still occurred.

I was hoping the recompiled libraries would solve that one. We didn't have time with the robot for generic programming much less systems debugging.

We have an upgraded RC returning in the mail either today or Monday just so we can explore the new RC issues. Let us know what you uncover.

[DjAlamose]

I’m not even sure if this is a "bug". The battery voltage in our batteries when fully charged is usually around 8 for the backup and 13.2 for the main battery. This is caused by the charger and is actually helpful because it makes the battery last longer. Im no big electrical person but i do know that the batteryies will not be exactly what they are rated to.

[Al Skierkiewicz]

Gene and Mark,
Does the OI report normal battery when not running the code? This is very interesting since the RC is on the hairy edge of reset at 8.2 due to the dropout restrictions of the internal regulator. Are you running the IFI backup battery charger circuit as well? When you measure the main battery are you at the battery or at the RC terminals?

[Mark McLeod]

Originally Posted by Al Skierkiewicz

Gene and Mark,
Does the OI report normal battery when not running the code? This is very interesting since the RC is on the hairy edge of reset at 8.2 due to the dropout restrictions of the internal regulator. Are you running the IFI backup battery charger circuit as well? When you measure the main battery are you at the battery or at the RC terminals?

All this is by my recollection, not directly from my test notes which are in our Robotics lab, so I may have forgotten some details.

Were not able to comparison test using the default IFI code or other variations. Ours was based on Kevin's streamlined camera code. We were running a mixture of drive train only and shooter/gimbal.

Normal battery voltage ups and downs were displayed on the OI until it suddenly went to an exact display of 8.2v.

We ran with the backup battery trickle charger circuit and charged backup battery. We did disconnected it for one of the tests, but we were looking at the sporadic controls problem and I don't remember if the 8.2v symptom also occurred when it failed. All while drivers and mechanics were simultaneously working out their problems of course. No servos or other drains on the backup battery, we even moved the camera-only power to an analog pin.

Measured main battery voltage at the battery was in the 11 to 12 volt range at the time under a light load (disabled). Under stress and very long running time the battery voltage was being driven down to 6 volts and the RC was transitioning to the backup battery fine.

[Gdeaver]

The Pic's will keep running below 5 volts unless IFI implemented the voltage watch dog. The RS232 however is very sensitive to voltage drop out witch might explain the printf. Could you have an intermitant power conection to the RC controler? Swap a different breaker too.

[Mike Bortfeldt]

We had the same issue with the OI displaying an 8.2/8.3 battery voltage (changed by the .1 volts when disabled). It was sporadic for the most part. One of the last times we had the issue, we were able to perform a number of code changes, download and attempt to isolate the problem. It seemed to be related to a particular (custom) code module we were using. We commented out all the code in the module, but left in the file variable list, and still had the issue. One by one, we removed the variables until we removed a specific structure and the problem went away (we had several variables of the same structure and some were still defined, so it wasn't the structure itself). After we thought we identified what was causing the issue, we started adding things back in until we were back to the original state of the module and couldn't recreate the problem (so was it the structure or did it just fix itself?). The only ideas we had at the time was that we shortened the variable section size such that the compiler placed the data in a different memory bank. It sure looked like some of the IFI variables were getting stomped (txdata, rxdata maybe?). After this, we switched to a spare 2004 controller we had and made the slight modifications necessary to get our code to run on that. We had no problems for the last week before ship using the old controller. I was really hoping that the new libraries were going to solve the problem, but it will be hard to tell with it being so intermittent. We have since gotten the 2004 controller upgraded by IFI and will try some testing on that this weekend.

One other item of note: the first thing we did when we saw the 8.2 voltage problem was disable all our user interrupts. We also deleted all the code from the InterruptHandlerLow to ensure that if we still had one active it would generate a red code error condition. It didn't help.

Mike

[eugenebrooks]

Originally Posted by Al Skierkiewicz

Gene and Mark,
Does the OI report normal battery when not running the code? This is very interesting since the RC is on the hairy edge of reset at 8.2 due to the dropout restrictions of the internal regulator. Are you running the IFI backup battery charger circuit as well? When you measure the main battery are you at the battery or at the RC terminals?

We measured our main battery voltage during one of these events and it was 12.6 volts at the power terminals of the RC.

We are not running the IFI backup charger circuit, we just keep our backup batteries swapped every match. It could very well be the backup battery voltage that is appearing on the user display, but this line of code is not in our program, we are using the user mode display to provide height information from the camera and rpm information, not the backup battery voltage.

As noted by others above, the problem seems related to some memory stomping that is going on and small changes to the code that move things around in memory cause the problem to come and go. We have seen wierdness on this front even when using a non-interrupt open loop code
for all controls. We have a "static int counter" in our main loop, used
to control printing every 40 trips. When we used

if(counter % 40 == 2) { printf(...); }

we lost our prints. When the counter was explicitly initialized
as "static int counter = 0;" the prints came back. The prints would
come and go with the "= 0" in the code to initialize the counter. Either
the compiler is not correctly initializing the static variables that are
not explicitly initialized, or the memory movement that might be caused
by the variable landing in a different segment, the one with explicit inits,
caused a memory stomping problem to go away. Remember, this
was with the new default code and simple open loop feedback control,
no use of timers or interrupts, beyond what is in the default code.

Given this experience, we went through the Kevin's camera code and
explicitly initialized all of the non-array static variables, and the resulting
binary locked up our RC with a programming light on, but it would not
load a program. Holding down the program button for the time period that
is normally takes to get the programming light allowed us to get a
the non-interrupt code back in place.

So, either the compiler or RC is not properly initializing implicitly
initialized static memory, or memory stomping is occuring even
with the default code... We are pretty skittish at changing the
code at this point, given that we need the robot working for matches
at the Portland regional, but given the fault that has appeared with the
static counter in the default code we may spend some time exploring
how this fault is affected by adding more variables before and after
the counter, to move it around in memory.

I would really like to be using the 2005 controller. We have some really
nice automatic aiming and ranging code that uses the camera, but we
can't trust that it will run reliably enough to use it in a match. The interrupt
based code on the 2006 controller would
have to get through an entire Thursday, while making code changes, without any of this kind of faults for us to trust it for a competition match.

Eugene

[Mike Bortfeldt]

Eugene,

I've been doing some testing on our recently IFI upgraded controller and have run into an interesting situation. It seems that if you set bit 7 of User_Byte3 (User_Byte3 = 128) that you can get the yellow "Low Main Batt." LED on the OI to turn on. I've tried using two completely different code projects with the same result. This definitely doesn't seem right and was wondering if you or anyone else could confirm that this happens on another 2006 controller before I try contacting IFI. (Tethered, no I/O attached, no backup battery)

Mike

[chris31]

We were getting this problem also. Unfortunantly it cost us 2 matches at the NASA/VCU regional in Richmond as we were unable to move! We didnt have alot of time to debugg the issue but we will post more as we find it out. When we were on tether it worked perfect, but when running through the competition port we had issues. We will post more as we find out.

EDIT: Al Skierkiewicz was asking some questions and we have answers. We were not running the backup battery charger circuit. The OI was reporting a voltage of 8.2, but the low battery indicater LED was not turning on. This stuck as us odd.

EDIT2: Also, this issue only arised after we went back to an older code base that was based on Kevin Watsons streamlined camera code.

[Sachiel7]

Yes, as Chris31 said above, this bug cost us two whole matches today at VCU.
We were quite frustrated in that we could not isolate the problem.

What the "bug" does is it returns a constant 8.2 volts on the OI, YET, the Low main bettery light does NOT cut on. In addition, the 8.2 volts does not change when you do manage to get systems running (It should drop with current draw).

We had interesting behaviour when our RC was running this bug. First off, our autonomous mode did not run. Second, pwm03 worked properly, but 04, 05, 13, and 15 stopped. Our relay based loader continued to function.

The thing about the bug is that it was very sparotic. It would randomly cut in or out, when you started up the bot, yet I don't think it ever changed while the bot was in operation.

-Sigh-
This bug cost us two whole matches when we finally had our robot and autonomous very functional.

Just to teams about to compete out there, keep your eyes on this one, it will kill matches....

[chris31]

I want to test our code with a new version of the camera code. Kevin Watson used interupts to handle serial port data, i beilive. And if i recall this is the only space. If i take that code out and dont use any interupts, will this still happen? We will just have to wait and see. Just a warning to all other teams, watch out. Its a match killer.

[Sachiel7]

Yes, if you want to see some video of this bug in action, watch us (1132) in these matches:
http://soap.circuitrunners.com/2006/...nia/va_071.wmv
http://soap.circuitrunners.com/2006/...nia/va_079.wmv

You might notics a few balls popping weakly out of our bot, because some systems were still working.
Robot reset button and OI Reset button did nothing.
Don't get caught by this guys...

[chris31]

After having this happen to us i am very interested in making sure that it doesnt happen to other teams. I am working on some code as we speak that would work the camera without the interupts Kevin used. Unfortunantly i do not have the bot, if someone would like to test code for me please PM me or say something here.

[Kevin Watson]

Originally Posted by chris31

I want to test our code with a new version of the camera code. Kevin Watson used interupts to handle serial port data, i beilive. And if i recall this is the only space. If i take that code out and dont use any interupts, will this still happen? We will just have to wait and see. Just a warning to all other teams, watch out. Its a match killer.

Just FYI, the camera code uses a serial port driver that teams have been using for over a year now without a problem. The camera, once initialized, only sends a few hundred bytes of information per second, which is easily handled by the PIC microcontroller.

-Kevin