Temperature sensitive controller? maybe 8.2 V related?

[ericand]

We have been running tests on a controller that has shown the 8.2V issue
and which has been getting intermittent code errors to try to determine if
or how our code is going wrong.

What we determined after much experimentation is that there is some sort
of hardware issue with the controller. We observed:

The code ran better when the controller was warm. We noticed that
code errors seemed to decrease when the controller had been
continuously operating for a period of time. We tested this by
cooling the controller and noticing that the incidence of code errors
went up. The cold controller would error immediately. The warm
one would error intermittently.

Older code which we thought didn't have problems was actually
seeing corruption in the data being sent to/from the OI. We noticed
this by seeing a correlation between joystick movement and feedback
LEDs that is not programmed into our code. With this in mind, we
looked at the feedback response with the code that has been
getting the code error and noticed that feedback LEDs that should be
lit continuously were blinking off intermittently.

Our theory is that the data segment that contains the txdata, rxdata, and the
status flag is somehow getting corrupted. Our map file shows this data is
located in gpr15 (above the stack? near the register access??).

We ran an experiment and changed the linker script by eliminating gpr15
(I commented it out, I'm not sure if that is the best solution). We then ran
the code on a cold processor and did not see any strange behavior. LEDs
that were supposed to be lit were lit and stayed lit. There were no strange
relationships between controls and LEDs. txdata etc are now located in gpr2
at 0x200 instead of gpr15 at 0xf00. I think I need to run more tests, but I
find these results interesting.

I now see that there are new linker scripts from Kevin and one of them marks
gpr15 as protected. Is there something known or suspected about data
integrity in gpr15? I don't have the controller here, but a quick recompile and
look at the map indicates that 18f8722beta.lkr moves the txdata into gpr2 like
the change I had made.

We need to run more tests, but I would really like to hear any comments on
our analysis.

B.T.W - we did run timing checks on our main loops including the process
data from master up() and we determined that our maximum loop time was
11ms with an average loop time less than 5ms so we are sure that we were
not getting the code errors from running too long.


Thanks for your help.

Eric Anderson - 1425 Wilsonville Robotics

[Eldarion]

Awesome work, guys! That makes a lot of sense. Maybe there is an intermittent internal connection in some of the processors?

Did you try heating up your RC (with a hairdryer or heat gun) and see if the bug was completely cured?

Or, put it in the refrigerator and see if the problem occurs all the time.

[ericand]

Originally Posted by Eldarion

Awesome work, guys! That makes a lot of sense. Maybe there is an intermittent internal connection in some of the processors?

Did you try heating up your RC (with a hairdryer or heat gun) and see if the bug was completely cured?

Or, put it in the refrigerator and see if the problem occurs all the time.

Actually we put it in a refrigerator for about 5-10 min when the error stopped happening and it came right back.

[chris31]

Originally Posted by ericand

Actually we put it in a refrigerator for about 5-10 min when the error stopped happening and it came right back.

Wow, thats crazy. Could you try and heat it up with a blow dryer or heat gun and test the errors then. Before i spend alot of time has anyone looked at the chip specs to see what its working tempurature is.

[ericand]

This looks like it is being confirmed by IFI. Stay tuned.

IFI has posted an update on their web site dealing with
the 8.2Voltage issue. Team 1425 feels that the issue that
we observed with the code errors and other random
behaviour is related (likely a different symptom of the same problem).

We are in contact with IFI and the IFI rep reported that
they reproduced the temperature sensitivity issue.

The solution posted on their site is listed as "temporary" so
I would expect future updates from them. Check out:

http://www.ifirobotics.com/docs/memory_problem_8722.pdf

[kaszeta]

Originally Posted by ericand

We are in contact with IFI and the IFI rep reported that
they reproduced the temperature sensitivity issue.

Well, I finally managed to use the temperature approach (using a large can of TFE "Office Duster" to cool down the RC) and this finally allowed me to recreate the 8.2V error in our clubhouse (it was doing it all the time in the pits, but was resisting my attempts to trigger it in the clubhouse). Replacing the linker script fixed it, as does warming up the RC.

I still think our 2004 RC (un-upgraded) is going in my bag for Palmetto in case this problem rears it's ugly head again.

[ericand]

I'm glad to hear you confirm what we observed.