2006 CMUcam2 Code

[Kevin Watson]

Originally Posted by nheft

We at team 533 are seeing two distinct intermittent startup error conditions that are unique to the frc_camera_s code. (haven't spent much time with the bells-and-whistles version). These are problems that do not occur with the IFI default code. The camera code we are using is the 1/11/2006 10:45 PM date stamp.

When the RC starts up correctly (about 90% of the time), the camera handler and servo tracker work perfectly. With the default_routine() call uncommented, we are also able to control other PWM putputs from the joysticks, etc.

The other 10% of the time we get one of two kinds of errors.

The first error is the "Code Error", similar to that reported by team 1216. The Code error only occurs at power-up, not after hitting a reset. When we see a Code Error, a robot reset clears the error.

The second error, far more disturbing than the "Code Error" condition, is one in which there is no code error but the data the user uP receives from the master is corrupted - for example Joystick 1 X and Y data fields are reversed. The disturbing thing about this condition is that everything looks normal on the RC LED's so we would not know the problem is happening until the match starts. Also, the RC does not recover from this case by pressing the reset button. Only cycling power (and waiting another 5 seconds if the backup battery is connected) clears the problem.
I disabled Tracking_Info_Terminal(), Camera_Handler() and Servo_Track() in Process_Data_From_Master_uP() and still get the startup errors.

I spoke to Tom Watson at IFI and he said that if we can't duplicate this with the IFI default code then it must be something in the camera code.

Any help would be appreciated. Thanks in advance.

I'd suspect that you've got a questionable robot controller. The addition of the camera code may be pushing executable code into an area of memory that may have access time issues. Call up IFI again and ask to speak with Mark, who is on the software side of the house. Hopefully he can arrange a swap and put your RC on a debugger to determine what's happening.

-Kevin

[Alan Anderson]

It sounds like the problem is with communication, and most of that is handled by the master processor. Reloading the master processor firmware is worth a try.

[nheft]

Thanks - I'll give 'er a try and let you know what happens.

Originally Posted by Alan Anderson

It sounds like the problem is with communication, and most of that is handled by the master processor. Reloading the master processor firmware is worth a try.

[nheft]

Tried reloading master.bin, to no avail.

Originally Posted by Alan Anderson

It sounds like the problem is with communication, and most of that is handled by the master processor. Reloading the master processor firmware is worth a try.

[nheft]

Our controller is en route to IFI as we speak. A swap was not possible; we're taking downtime and focusing on mechanical issues until we get our controller back.

Before sending it back I ran some more tests and made an interesting observation - I placed a light duty Pep Boys automotive switch in series with the HI-AMP circuit breaker and used the switch to turn the controller on and off. Still seeing intermittent failures but significantly less often than when powering up and down with the breaker. These observations were made with the 7.2 backup battery out of the equation. The only difference I can think of is the contact bounce characteristics of the switch versus the HI-AMP breaker. I looked at the turn-on transient of the breaker and the switch on a scope. They each exhibit about 5 ms of contact bounce before finally settling out to +12V. Tom at IFI thinks what I saw was coincidence. Wierd problem. Time will tell what the cause is.

[Anime-niac_2.9]

the camera code is being a pain. tell us kevin, what could we edit so that the camera can scan smoothly instead of quick jolt-stop-quick jolt-stop, and so forth? just what was the purpose of having the camera being able to look directly at the ground.

[Cuog]

you can edit the tracking portion of tracking.c I changed ours to track smoothly and it first tracks where i tell it to(where i predict the light) then it searches a wider pattern until it finds the light

[drinkdhmo]

We are having the same problems with the camera_s code as team 533. Except, the code fault on powerup occurs almost 95% of the time, and this is fixed with a reset. Also, instead of the p1_y and the p_1x being switched, the joystick's y axis is being read as p1_x in the controller and the joystick's x-axis and throttle are unresponsive. The joystick's buttons are also giving very flaky values, even when not pressed. We are watching these values with a printf in the Process_Data_From_Master_uP function with an otherwise fresh copy of the camera_s code that is dated 1/14/06.

[nheft]

When we hit the reset button and the code error goes away, the p1_y still gets read as p1_x. Do you see the same?

It makes a difference how you power up the controller. We placed a toggle switch between the fuse panel and the RC. Sure enough, the power-up fault occurs much less often (but still happens occasionally) when we use the switch to turn the controller on. I think the difference is in the contact bounce of the switch vs the circuit breaker.

Your description of the joystick's p1_y being read as p1_x is exactly what we're getting; my previous statement that x and y were reversed was inaccurate.

I sent our controller back to IFI and they claim they cannot duplicate the problem at IFI.

I spoke to Mark Lambert IFI's RC digital/software guru. He can't identify a mechanism that would explain what we're seeing. Since it does not happen with the IFI default code, they will not spend much time investigating. The only software they "support" is the IFI default code.

Mark did give me some information that might help. The "reserved" word in the rxdata structure always contains the number 12, which is the firmware revision number for the master uP. When the rxdata gets misaligned (p1_y read as p1_x, etc), the reserved word will probably be wrong too. I can use this to detect the error, go into an infinite loop, and force the RC to crash. This way we can tell by looking at the RC LED's that something is wrong and try recycling power. Yes, it's just a band-aid but we don't have time to dig in and find the root cause.

[nheft]

Kevin,

Here is a follow up to my initial post on startup errors.

Suspecting a questionable RC, I did the following:

1. Loaded unmodified hex file from the latest frc_camera_s release
2. Verified that we still see the intermittent code error
3. Sent the controller back to IFI.

Tom Watson and Mark Lambert cycled power on the controller at IFI 150 times (75 with the backup battery and 75 without the backup battery) and they claim they can't duplicate the problem.

Since then, another team, 1566, has reported the exact same problem on this forum.

I have noticed that, strangely enough, when using a toggle switch instead of the HI-AMP kit breaker to power the RC up, the error becomes far less frequent.

Any ideas?

Thanks in advance.

[Mark McLeod]

You might try dumping the entire rx packet to a file so you can do a full compare of good packets against the bad packets. There may be further clues in what's getting corrupted. Since it may be one dropped value at the start causing a shift in all the other packet values (possibly double 255's?).
It would be handier if you have a OI pot/switch box to preset known values to make the proper contents more readily identifiable.

I'll see if I can reproduce the errors you guys are seeing on our 2006 RC this evening. That'll give it third party confirmation.

[Kevin Watson]

Originally Posted by nheft

Kevin,

Here is a follow up to my initial post on startup errors.

Suspecting a questionable RC, I did the following:

1. Loaded unmodified hex file from the latest frc_camera_s release
2. Verified that we still see the intermittent code error
3. Sent the controller back to IFI.

Tom Watson and Mark Lambert cycled power on the controller at IFI 150 times (75 with the backup battery and 75 without the backup battery) and they claim they can't duplicate the problem.

Since then, another team, 1566, has reported the exact same problem on this forum.

I have noticed that, strangely enough, when using a toggle switch instead of the HI-AMP kit breaker to power the RC up, the error becomes far less frequent.

Any ideas?

Thanks in advance.

I think we can go on the assumption that it's not caused by the camera code directly. The only thing that immediately comes to mind is some kind of race condition where the processor comes out of reset before the oscillator is stabilized. If I get a chance, I'll call IFI today and see what can be done about this. As an experiment, you might try holding the processor in a reset state while powering up. Do it enough times to get real statistics.

-Kevin

[nheft]

If you can replicate the problem on your RC then you will be the fourth team to my knowledge that has seek the problem first hand.

If you can try just loading the hex file from frc_camera_s (which can be found on http://www.kevin.org/frc) and try a few power cycles and let me know what you find it would be a tremendous help. I'm going to try the full packet dump idea this afternoon at the school.

Originally Posted by Mark McLeod

You might try dumping the entire rx packet to a file so you can do a full compare of good packets against the bad packets. There may be further clues in what's getting corrupted. Since it may be one dropped value at the start causing a shift in all the other packet values (possibly double 255's?).
It would be handier if you have a OI pot/switch box to preset known values to make the proper contents more readily identifiable.

I'll see if I can reproduce the errors you guys are seeing on our 2006 RC this evening. That'll give it third party confirmation.

[nheft]

Once the corrupt data problem (p1_y being read as p1_x, etc), the reset button does not clear the problem; nothing works except cycling the power off and back on. I have not tried powering up with the reset button pressed as you suggested; I'll try that idea tonight at the school.

Another thing I will try is to put a cheap and dirty delay loop ahead of IFI_Initialization to stall the user processor for a few tenths of a millisecond after a reset.

When I post an observation here at Delphi it is typically based on a collection of 50 trials or more.

Originally Posted by Kevin Watson

I think we can go on the assumption that it's not caused by the camera code directly. The only thing that immediately comes to mind is some kind of race condition where the processor comes out of reset before the oscillator is stabilized. If I get a chance, I'll call IFI today and see what can be done about this. As an experiment, you might try holding the processor in a reset state while powering up. Do it enough times to get real statistics.

-Kevin

[drinkdhmo]

We have explored the problem a little more fully today and have found a few more details:

Powering up the robot with the reset button depressed, then releasing it, still results in a code fault.

The processor results in a code fault only when the controller has been off for more than 5 seconds after running for more than a few minutes. We did some extensive testing and timing and a power cycle (with the breaker, not the reset button). When the processor has not been on as long, more time powered down is required to reproduce the code fault. We could not reproduce the code fault at all by holding down the reset button for an extended period of time (about a minute).

Along with joystick 1's y-axis being mapped to p1_x, we have also noticed that joystick 3's x-axis is being mapped to p1_y. We have not yet explored all of the oi input mappings, but we will post it when we do.

We also found that when we powered up the rc without a radio modem or tether connected, the program does not fault. When we reconnect the radio modem the program will fault again, but only after the processor has been powered down for a varrying period of time.

Whether the code faults on startup or not, the OI input mappings are still messed up.

This is all we have found so far. I will continue our exploration of the problem tomorrow and post anything new.