Camera does not search in autonomous mode after reset

[Keith Watson]

We encountered a problem for the first time today at the regional. It only happens in autonomous mode. Take the robot to the playing field, plug in the backup battery, power on the main battery, press reset on the controller, the camera does not initialize.

I've read other people have encountered the same problem but have not seen anyone have an actual fix.

The camera code is correctly called from autonomous mode and works if the robot is put into user mode then back into autonomous mode (which you obvisously cannot do on the field). We even did this in the pits and left the robot powered on till it was put on the playing field. It worked that time.

We have since found a spot in the code which is called in autonomous mode right before the inner while loop. We are going to try to reset the camera intialization variable there and test it during the finals.

The behavior is the same as if a controller reset does not call the program's static initializers in autonomous mode!

[Bharat Nain]

Originally Posted by Keith Watson

We encountered a problem for the first time today at the regional. It only happens in autonomous mode. Take the robot to the playing field, plug in the backup battery, power on the main battery, press reset on the controller, the camera does not initialize.

I've read other people have encountered the same problem but have not seen anyone have an actual fix.

The camera code is correctly called from autonomous mode and works if the robot is put into user mode then back into autonomous mode (which you obvisously cannot do on the field). We even did this in the pits and left the robot powered on till it was put on the playing field. It worked that time.

We have since found a spot in the code which is called in autonomous mode right before the inner while loop. We are going to try to reset the camera intialization variable there and test it during the finals.

The behavior is the same as if a controller reset does not call the program's static initializers in autonomous mode!

We have had some weird problems on the field all day too but we COMPLETELY unplug all our batteries and then check all our electrical connections and put it back in. It seemed to have worked fine after that.

[Keith Watson]

We would completely unplug our batteries also. We could reproduce the problems in the pits by setting our OI to autonomous. During the matches today the camera did initialize so resetting the variable I mentioned does work.

[Uberbots]

Originally Posted by Keith Watson

We would completely unplug our batteries also. We could reproduce the problems in the pits by setting our OI to autonomous. During the matches today the camera did initialize so resetting the variable I mentioned does work.

Hmm...
You know that section in the User_Autonomous_Code function, right before the "while (autonomous_mode)" loop (and right after all of the PWM values are reset)?
Our team had been experiencing problems with that, and so (in my hacking nature...) I had put the Initialize_Camera function in that section. I don't know why, but it helped, and now the camera always initializes when Autonomous mode starts (or is invoked).

also, while you are executing the Autonomous loop, are you sure you are calling the function "Camera_Handler()"?

[Keith Watson]

Originally Posted by Uberbots

Hmm...
You know that section in the User_Autonomous_Code function, right before the "while (autonomous_mode)" loop (and right after all of the PWM values are reset)?
Our team had been experiencing problems with that, and so (in my hacking nature...) I had put the Initialize_Camera function in that section. I don't know why, but it helped, and now the camera always initializes when Autonomous mode starts (or is invoked).

That is the same section of code where I added

tracking_initialized = 0;

Originally Posted by Uberbots

also, while you are executing the Autonomous loop, are you sure you are calling the function "Camera_Handler()"?

Yes. Our programmer found camera searching was not working in autonomous and he and I tracked down why and added both "Camera_Handler()" and "Servo_Track()".

The behavior I described as if the static initializers are not being called under certain conditions is also happening in the recent thread The 8.2 (or 8.3) Battery Voltage Bug.

[Keith Watson]

Come to think of it, we had "intermittent" camera initialization problems from day one. Most of the time we assumed it was from a low backup battery so we were constantly swapping those in and out. Camera initialization always failed for autonomous mode when the competition port was connected. We struggled a long time with assuming we were not starting up the robot the same way every time. It was not until Friday night at the regional when we were finally able to consistently reproduce the problem in the pits. That was when we decided to try a non-static initializer.

And I no longer have access to the robot to write a test to prove the static initialization conditions.

The static initializater for camera initialization is in Kevin's bells and whistles camera code.

[kaszeta]

Originally Posted by Keith Watson

Come to think of it, we had "intermittent" camera initialization problems from day one. Most of the time we assumed it was from a low backup battery so we were constantly swapping those in and out.

Same here...

And I no longer have access to the robot to write a test to prove the static initialization conditions.

I do, I have our teams 2006 RC sitting in front of me (since I'm going to send it to IFI this week). If I have a spare moment I'm going to write some test code to check on this, as well as read over the 8722 manual and errata (I had a similar issue to this with another 18F chip on a project at work, so I wouldn't be surprised).

Oy, amazing how repeated testing of a bot at home can show things to be surprising robust, but fail completely at competition like clockwork...

[steven114]

It seems that the problem can manifest itself as a messed up state machine - where there is the large switch() statement with CASE_ONE through CASE_WHATEVER, the state becomes set to zero accidentally so it falls through every time and does nothing.

[Eldarion]

Originally Posted by Keith Watson

We encountered a problem for the first time today at the regional. It only happens in autonomous mode. Take the robot to the playing field, plug in the backup battery, power on the main battery, press reset on the controller, the camera does not initialize.

I've read other people have encountered the same problem but have not seen anyone have an actual fix.

The camera code is correctly called from autonomous mode and works if the robot is put into user mode then back into autonomous mode (which you obvisously cannot do on the field). We even did this in the pits and left the robot powered on till it was put on the playing field. It worked that time.

We have since found a spot in the code which is called in autonomous mode right before the inner while loop. We are going to try to reset the camera intialization variable there and test it during the finals.

The behavior is the same as if a controller reset does not call the program's static initializers in autonomous mode!

I got to thinking about this. Is this the reason IFI has the second "initialization" section directly before the autonomous while loop? Would it be a good idea just to bypass IFI's initialization calls altogether and make your own initialized / not initialized flag that, when zero, calls the init routine and sets the flag to 1?

I distinctly recall a similar problem in our custom code last year, with the robot acting completely bezerk on the field, yet in the pit with the dongle, it worked perfectly. I had heard a rumor about the Arena Controller forcing some king of sync with the OI and RC; is there anything to justify this rumor? It might explain a lot of the "it worked perfectly until I put it on the field" problems.

Would this issue be reproducable on a 2004/2005 RC, since it seems to be a problem with IFI's default code?

Just some ideas...

[Keith Watson]

Originally Posted by steven114

It seems that the problem can manifest itself as a messed up state machine - where there is the large switch() statement with CASE_ONE through CASE_WHATEVER, the state becomes set to zero accidentally so it falls through every time and does nothing.

Neither our code nor tracking.c uses switch statements. Camera.c does though and does not have default cases. Bad idea. I never write switch statements without a default case even if it just prints an error message.

[bear24rw]

Originally Posted by Keith Watson

I never write switch statements without a default case even if it just prints an error message.

What would happen if you didn't have a default case?

[Keith Watson]

Originally Posted by Eldarion

I got to thinking about this. Is this the reason IFI has the second "initialization" section directly before the autonomous while loop? Would it be a good idea just to bypass IFI's initialization calls altogether and make your own initialized / not initialized flag that, when zero, calls the init routine and sets the flag to 1?

The only initialization I see is giving all outputs a default value. Not only is there a comment explaining why, but the user code gets called once on startup even when in autonomous mode. So forcing outputs to off is a good idea.

Originally Posted by Eldarion

I distinctly recall a similar problem in our custom code last year, with the robot acting completely bezerk on the field, yet in the pit with the dongle, it worked perfectly. I had heard a rumor about the Arena Controller forcing some king of sync with the OI and RC; is there anything to justify this rumor? It might explain a lot of the "it worked perfectly until I put it on the field" problems.

Would this issue be reproducable on a 2004/2005 RC, since it seems to be a problem with IFI's default code?

Just some ideas...

Thanks for the ideas. In the rush at the Regional we never had the time to sit down, develop a test plan, and methodically track down the problem. But we did find a sequence of events where we could reproduce the problem in the pits. Our code was just not instrumented to diagnose the problem.

Today I developed a test plan involving: using the default frc_camera_2 code, adding camera tracking to autonomous, and instrumenting the camera and tracking initialization variables to show their state on OI lights. My team contact is on the other coast for 2 weeks so I may not be able to arrange test time for a while.

Unfortunately our regional is over and we are done for the year. Our strategy was heavily weighted towards scoring during autonomous. All the pieces were in place but the camera never worked during autonomous.

[Keith Watson]

Originally Posted by bear24rw

What would happen if you didn't have a default case?

In general, if you end up with an invalid state you will have unpredictable results. Specifically with regards to camera.c, I have looked at the code but cannot predict what might happen. The first walk through looks like it should work. But I have not worked through the possibilities to see if it is possible to have an uninitialized state.

[Kevin Watson]

Originally Posted by Keith Watson

Neither our code nor tracking.c uses switch statements. Camera.c does though and does not have default cases. Bad idea. I never write switch statements without a default case even if it just prints an error message.

Sure, on large, complex systems with lots of resources you'd instrument the code or HDL to catch such cases. Instrumenting a very simple state machine that cannot (by inspection) have an invalid state, in a resource constrained system where code readability is of paramount importance, doesn't make sense to me. If the static state variable isn't getting initialized at boot time, a wonky state machine is the least of your worries .

[eugenebrooks]

Originally Posted by Kevin Watson

Sure, on large, complex systems with lots of resources you'd instrument the code or HDL to catch such cases. Instrumenting a very simple state machine that cannot (by inspection) have an invalid state, in a resource constrained system where code readability is of paramount importance, doesn't make sense to me. If the static state variable isn't getting initialized at boot time, a wonky state machine is the least of your worries .

Now I know what that third LED relay light is for, we found a use for
all of the other lights on the OI. We will label it the "totally
screwed" light. It is best to know this condition as early as possible.

Sounds like the default action for the switch statement to me...