The 8.2 (or 8.3) Battery Voltage Bug

[Kevin Watson]

Originally Posted by chris31

I want to test our code with a new version of the camera code. Kevin Watson used interupts to handle serial port data, i beilive. And if i recall this is the only space. If i take that code out and dont use any interupts, will this still happen? We will just have to wait and see. Just a warning to all other teams, watch out. Its a match killer.

Just FYI, the camera code uses a serial port driver that teams have been using for over a year now without a problem. The camera, once initialized, only sends a few hundred bytes of information per second, which is easily handled by the PIC microcontroller.

-Kevin

[chris31]

Thank you for the information. Outside of the interupt driven code you wrote we did not use any interupts this year. I am trying to debug the issue to save teams from having this happen to them but all i have is info from debugging today, the last day of our regional, the code we used, and some video of the bot. IFI has recognized a problem with the interupts and that was why i was quick to jump to that conclusion. I wish i had the RC with me to test more but i dont. I havent been able to find out why this would just start happening. The code we were running when we got this error was your camera code with a modified default routine. Sachiel7 can confirm that for us. Sachiel7, if you have any more info please tell us.

EDIT: gobeavs, we have had that work also. That is a temporary solution. It doesnt solve the root of the issue which is what we would really like to happen. Can you supply more info on what the code you were running was.

[gobeavs]

Originally Posted by chris31

EDIT: gobeavs, we have had that work also. That is a temporary solution. It doesnt solve the root of the issue which is what we would really like to happen. Can you supply more info on what the code you were running was.

We are running code based off of Kevin's camera code, I think v1.

[Mike Bortfeldt]

I was just lucky (or unlucky) enough that this bug kicked in while I was bench testing. It seems that the 8.2 battery voltage is really the reading of the p4_wheel. As you move the joystick 4 wheel, the OI battery voltage changes. The p3_wheel is mapped to the backup battery voltage (as shown from the data in the dashboard). the p1_x axis now maps to the joystick #3 switches. I can't say exactly what works and what doesn't as I was just reading what I was sending back from the RC to the OI from the dashboard and didn't really map all the joystick data to the user_bytes/pwm values. A reset doesn't work, but a full power down did (although I know that it doesn't always from previous testing). I do know that one of the joystick ports seemed to be mapped to the "disable" flag (not sure which one), so it's quite likely that the "autonomous" flag was also messed up.

Mike

Correction - Rather than the p3_wheel & p4_wheel, it is actually the p3_aux and p4_aux.

[chris31]

Originally Posted by Mike Bortfeldt

I was just lucky (or unlucky) enough that this bug kicked in while I was bench testing. It seems that the 8.2 battery voltage is really the reading of the p4_wheel. As you move the joystick 4 wheel, the OI battery voltage changes. The p3_wheel is mapped to the backup battery voltage (as shown from the data in the dashboard). the p1_x axis now maps to the joystick #3 switches. I can't say exactly what works and what doesn't as I was just reading what I was sending back from the RC to the OI from the dashboard and didn't really map all the joystick data to the user_bytes/pwm values. A reset doesn't work, but a full power down did (although I know that it doesn't always from previous testing). I do know that one of the joystick ports seemed to be mapped to the "disable" flag (not sure which one), so it's quite likely that the "autonomous" flag was also messed up.

Mike

Interesting, Ill have a look. We decided early today to use are port 4 wheel for launcher wheel PWM. This could be part of the problem. Ill try and have that check out.

[chris31]

Originally Posted by awawadad

Uncalled for noobs

Maybe im just a little stressed from competition today, but you just joined today and made 5 rude posts that are bettering the community in anyway. If you dont stop i exspect to see an IP ban soon.

[EricS-Team180]

Originally Posted by Mike Bortfeldt

.....from page 2.....
...As you move the joystick 4 wheel, the OI battery voltage changes. The p3_wheel is mapped to the backup battery voltage (as shown from the data in the dashboard). the p1_x axis now maps to the joystick #3 switches. ...

Correction - Rather than the p3_wheel & p4_wheel, it is actually the p3_aux and p4_aux.

This doesn't directly relate to these microprocessors, but...
This reminds me of something I once encountered with PowerPC's, running VxWorks OS.
It had to do with how you saved the data registers when a function was
interrupted by a function with higher priority.
You could request that the interrupted functions's registers be saved as an "integer save" or a "floating point save". If you used an "integer save" and you had any floating point registers that you were using, your saved data was
corrupted.......a very low level and weird bug.

So...
How are the registers saved on the User proc when it is interrupted by the master proc for data i/o? A pointer hack, overflow or corrupted save could certainly be changing the apparent mapping of the i/o data. And, it could be reproducible on any of the pic FRC's (I believe ElDarion reproduced the problem on a 2k4)

...just my 2cents

[eugenebrooks]

Originally Posted by Kevin Watson

Just FYI, the camera code uses a serial port driver that teams have been using for over a year now without a problem. The camera, once initialized, only sends a few hundred bytes of information per second, which is easily handled by the PIC microcontroller.

-Kevin

The problem appears to be some form of memory stomping, in the IFI default
code, or in the controller chip itself. I don't believe that it is a bug in the
camera code per-se, although the use of the camera code increases the
severity of the problem. We backed out of all of our interrupt/timer based
control at the Portland regional in response to this problem as there was
no way we could keep our robot running otherwise. We have seen problems
in very simple code. Below are the only lines of custom code, using
the recently published default code.

At one point whether or not the periodic printf statements appeared on
he download console depended upon whether or not we had
static int counter;
or
static int counter = 0;
for the declaration of "counter". This problem comes and goes over
a day, in the morning it is consistent but when attempting to reproduce
it with the same code in the afternoon one can't after several tries.
I am presuming that the memory location
of "counter" moves between the implicitly initialized segment, from the
explicitly initialized segment, with the declaration change and this changed
the arrangement of memory. Whether or not memory is actually getting
stomped is something a little more random in the controller.

An option question is, given the problems with the 2006 controller,
whether or not FIRST would be willing to allow the use of the 2005
controller at regional events. We have not seen any of these problems
on the 2005 controller.

Eugene



/************************************************** *****************************
* FUNCTION NAME: Process_Data_From_Master_uP
* PURPOSE: Executes every 26.2ms when it gets new data from the master
* microprocessor.
* CALLED FROM: main.c
* ARGUMENTS: none
* RETURNS: void
************************************************** *****************************/

struct rpmpower {
int rpm;
int power;
};
/* This table is constructed by using the manual RPM control to set
RPM values manually and then reading off the required power for that RPM.
*/
#define RPMPOWERTABLESIZE 9
struct rpmpower rpmpowertable[RPMPOWERTABLESIZE] = {
1300, 160,
1400, 161,
1500, 167,
1600, 168,
1700, 170,
1800, 177,
1900, 181,
2000, 189,
2100, 193
};
int RPM2Power(int rpm) {
int RetPower;
int i;
if(rpm <= rpmpowertable[0].rpm) return rpmpowertable[0].power;
if(rpm >= rpmpowertable[RPMPOWERTABLESIZE-1].rpm) return rpmpowertable[RPMPOWERTABLESIZE-1].power;
for(i=0; i<(RPMPOWERTABLESIZE-1); i+=1){
if(rpmpowertable[i].rpm < rpm && rpmpowertable[i+1].rpm >= rpm){
RetPower = rpmpowertable[i].power + ((rpm - rpmpowertable[i].rpm) * (rpmpowertable[i+1].power - rpmpowertable[i].power)) / (rpmpowertable[i+1].rpm - rpmpowertable[i].rpm);
break;
}
}
return RetPower;
}


void Process_Data_From_Master_uP(void)
{
int Target;
int BallWheelPower;
static int counter; /* Does not work if not initialized to zero! */

Getdata(&rxdata);

/* Control of the drive motors.
*/
if(rc_dig_in08 == 1) {
pwm01 = p1_y;
pwm02 = p3_y;
pwm03 = p1_y;
pwm04 = p3_y;
}
else {
pwm01 = STOP;
pwm02 = STOP;
pwm03 = STOP;
pwm04 = STOP;
}

/* Pan Control
*/
if(p3_sw_trig == 1 && rc_dig_in12 == 0) {
pwm06 = STOP + 20;
}
else if(p2_sw_aux2 == 1 && rc_dig_in11 == 0){
pwm06 = STOP - 12;
}
else{
pwm06 = STOP;
}
if(rc_dig_in06 == 0){
pwm06 = STOP;
}

/* Ball lift control.
*/
if(p2_sw_top == 1 && rc_dig_in07 == 1) {
pwm05 = 254;
}
else if (p2_sw_trig == 1 && rc_dig_in07 == 1){
pwm05 = 1;
}
else {
pwm05 = STOP;
}

/* Ball wheel speed control.
*/
Target = (int)1200 + (int)3 * (int)p4_x;
BallWheelPower = RPM2Power(Target);
if(rc_dig_in10 == 0) {
pwm07 = STOP;
}
else {
pwm07 = BallWheelPower;
}

/* Ball shooter fire control
*/
if(p2_sw_aux1 == 1 && rc_dig_in09 == 1) {
pwm08 = 254;
}
else {
pwm08 = STOP;
}


if(counter % 40 == 2) {
printf("p1_y = %d, p3_y = %d, BWP = %d, TargetRPM = %d, p4_x = %d\r",
(int)p1_y, (int)p3_y, (int)BallWheelPower, (int)Target, (int)p4_x);
}

counter += 1;

if (user_display_mode == 0) {
if(rc_dig_in12 == 1) { /* Right pan limit. */
Switch1_LED = 1;
}
else {
Switch1_LED = 0;
}
if(rc_dig_in11 == 1) { /* Left pan limit. */
Switch2_LED = 1;
}
else {
Switch2_LED = 0;
}
}
else {
User_Mode_byte = backup_voltage * 10;
}

Generate_Pwms(pwm13,pwm14,pwm15,pwm16);

Putdata(&txdata); /* DO NOT CHANGE! */
}

[devicenull]

What happens if you move the "static int counter" outside of the Process_Data function?

Also, you may want to add the "rom" keyword to the rpmpowertable statement. It then becomes one less thing that could be overwriting other variables (As it would be stored with the actual program, rather then other variables)

[chris31]

I will continue to be a "Graciuos Professionalist" and not make any comments. Chief Delphi is a great community and flaming rarely occurs. Please do not make comments like that to me and others that spend our time trying to help others.

EDIT: D.J. Fluck, thank you for removing his post.

[kaszeta]

Originally Posted by eugenebrooks

We are using a combination of Kevin Watson's camera code, using only the sensor reporting, not the servo functionality, and a timer/sensor interrupt to regulate the speed of our ball shooting wheel. We see the camera go off line sporadically, with some of the printf() statements ceasing to appear on the upload line. When this happens, the battery voltage shown on the OI is 8.2 (or 8.3) even though the voltage of the main battery is just fine.

Our robot died three times during autonomous at BAE, and this was one of the symptoms at the time (main battery voltage reading as 7.75 volts on the OI). Eventually, the IFI rep requested that we (a) double-check our code (see other thread), and (b) use a loaner RC.

Interestingly, when I was double-checking the code after the first autonomous failure (still on our original 2006 RC), one of the first things I did is revert to a code version that doesn't use any of the camera code (but does use Kevin's ADC code), and we still had this problem. We also saw it on one of the loaner RC's (an upgraded 2004) as well, using both our code and the most recent default code to which we had added a single printf() reporting the battery voltages. The final loaner RC didn't show any problems .

So I've seen this problem on both our original 2006 RC *without* the camera code (using a fresh copy of Kevin's Gyro code with our default routine and autonomous added), and even on one loaner RC with default code (although other teams had reported problems with this loaner RC). So I'm guessing it's something with the interrupt handler (since we're using Timer 2, and the two sets of serials interrupts)? The serial code is common between the camera code and the gyro code.

I'm wondering if there is some sort of memory stomping happening, because I've also seen some random bits flicking on the OI status leds (the pwm and relay leds; we had disabled the default routine LED feedback on these to make a little bar graph showing how good the camera lock was, but after we pulled the camera I pulled the code for this as well, and we still would see random bits flicker in there).

I'm interested in solving this, since we had three matches that we lost basically since we died during autonomous (and were non-functional the rest of the match), and a fourth in which we competed with a 100% dead bot (since we pulled the RC to replace it with a loaner, but the first loaner was DOA, and we didn't have time to put the original RC back in). We still got five points with our bot that round (an alliance bot pushed us up on the platform...), but considering that we have a really good ball launcher, it rather stank that we couldn't run most of the time.

Unlike most teams, I now actually have the 2006 RC in my possession (a replacement is coming from IFI), although it's going to IFI this week once I get an RMA. Anything I should try checking with it?

In any case, if someone would like a copy of our code (that showed the voltage bug on two different RCs), or has some ideas, I'd like to know.

[Kevin Watson]

Originally Posted by kaszeta

...We also saw it on one of the loaner RC's (an upgraded 2004) as well, using both our code and the most recent default code to which we had added a single printf() reporting the battery voltages. The final loaner RC didn't show any problems...

So I've seen this problem on both our original 2006 RC *without* the camera code (using a fresh copy of Kevin's Gyro code with our default routine and autonomous added), and even on one loaner RC with default code (although other teams had reported problems with this loaner RC). So I'm guessing it's something with the interrupt handler (since we're using Timer 2, and the two sets of serials interrupts)? The serial code is common between the camera code and the gyro code.

When you say "default code", do you mean the default code from IFI's website or do you mean code from my website?

-Kevin

[kaszeta]

Originally Posted by Kevin Watson

When you say "default code", do you mean the default code from IFI's website or do you mean code from my website?

A fresh copy that I downloaded from IFI's website during the competition (http://www.ifirobotics.com/docs/frc-code-2-28-2006.zip). However, that RC was suspect (every once in a while the printf's would produce garbage).

[Keith Watson]

Originally Posted by eugenebrooks

...Either the compiler is not correctly initializing the static variables that are not explicitly initialized, or the memory movement that might be caused by the variable landing in a different segment, the one with explicit inits, caused a memory stomping problem to go away.

The compiler creates a section of code in the binary where all of the static initializers are called. I have always assumed any sort of processor reset would call this block of code before calling main().

Originally Posted by gobeavs

That bug hit us once and disabled us for a round at PNW, but we have found a way to avoid it. When we turn our robot on at the beginning of the round we have someone at the OI looking at the battery voltage to give us a thumbs up or down depending on whether it is showing ~12 or 8.2. If we get 8.2 we just power cycle (not just reset, because that screws up the camera). It saved us at least once later on.

We encountered behavior where the symptom is that the static initializers are not being called by a reset under certain conditions. See the recent thread Camera does not search in autonomous mode after reset for a full description of those conditions.

[Mike Bortfeldt]

I've been able to successfully cause the 8.2 battery voltage bug to occur on demand by sequentially downloading 2 different versions of code (both based off the default code). I haven't narrowed down the exact cause, but I will see what I can find out tonight and call IFI tomorrow. With a way to reproduce the problem, they will hopefully be able to come up with a fix or workaround.

Mike