[FTC]: Playing Forensic Roboticist.

[PhilBot]

I love trying to find out why things work (or don't work).

Most teams were frustrated at the MD Regional (the only one I attended) at least once by having dead controls at the start of Teleop. There didn't seem to be any reson for this as the FMS indicated that the connection was good, and the programs were running.

I've been doing a lot of code tweaking over the last day, and I've seen something which may shed a light on this.

I was running mock matches with the bot up on blocks, and I was just running and re-running the same code looking for a problem.

At least 3 times in a sequence of 10 runs, the program ran Autonomous, and then when when it was meant to transition to Teleop, it stopped the Auto program and then failed to restart the NXT in Teleop. It just sat there with the "ready to run" screen up.

What was more disturbing was that the FMS system indicated that the NXT was "running", whatever that means.

So to all intents and purposes it "appeard" that everything was fine, but there was no way to drive the robot.

At the end of teleop, the match was stopped but if you looked at the NXT then, everything seemed OK (since the program SHOULD have been stopped at that point).

So I suspect that if the real FMS software works the same way, there may have been many instances where the USB controllers and Bluetooth were just fine, but there was no Teleop program running. (Just frustrated kids wiggling controls)

I wonder if this problem could be minitigated by sending several "RUN TELEOP" commands to the NXT. Once is bound to catch.

Now I'm thinking of a way to indicate that the TELOP program is running on our NXT, so we can allert the field crew that we have a problem that's not Code related. Maybe a flashing red light sensor?????

Anyone else see this while testing?

[ptan]

Originally Posted by PhilBot

At least 3 times in a sequence of 10 runs, the program ran Autonomous, and then when when it was meant to transition to Teleop, it stopped the Auto program and then failed to restart the NXT in Teleop. It just sat there with the "ready to run" screen up.

What was more disturbing was that the FMS system indicated that the NXT was "running", whatever that means.

Interesting... Were you running a full field (i.e. 4 robots?). I have not yet seen this problem in our team's test field, but we have been only running our single robot with the FMS. We did experience this at the Ontario tournament along with several bluetooth issues.

I'm going to try to set up a similar test bed as yours, but with just 4 NXT controllers and the FMS to see if I can recreate the failure rate (3 out of 10) that you reported. Can you tell me if you are using Windows XP or Vista? And what service pack level are you using?

Like yourself, I'm also interested in getting down to the bottom of these mysteries.

[PhilBot]

Originally Posted by ptan

Interesting... Were you running a full field (i.e. 4 robots?).

I was running a minimal system, with 1 Robot and one Controller.

In this instance I'm running Windows XP Professional 2003 SP3

[jbbjjbt]

Phil,
We have never seen what you describe, and we've run the fms hundreds of times trying to get the autonomous to do what we tell it. Teleop always runs fine after autonomous. We see the opposite alot, where autonomous doesn't run on the first try. Removing the battery etc seems to make autonomous more reliable but is a big pain when you are running mutlitple times to tweek your autonomous.

Also we've been in three competitions (VA, DE, and MD) with two robots and haven't seen it in competition. At Delaware, many other teams had issues which they now claim were due to faulty USB hubs. We did loose control once near the end of a run when a robot ran into the wall, which supports their faulty usb hub theory.

Does the nxt look like it's running? Have you tried adding a lego motor and see if it responds? Have you tried with the Logetech controllers directly into the computer rather versus in a USB hub or vise versa. We seem to be having an issue on our laptop with the USB ports getting loose due to all the plugging and unplugging.

Jon

[PhilBot]

Originally Posted by jbbjjbt

Phil,
Does the nxt look like it's running?
Jon

In the example I just descibed, the NXT was clearly not running.

It's the old story though, the more debugging you put in, the harder it is to know whether you are effecting the operation of the program.

To make it obvious that the program was running, I added a light sensor, and turned on/off the "illumination" to make it act as a signal light. One flash per second for Auto, and two flashes flashes per second for Teleop.

It make it pretty clear when the program was running/stopping.

Just this morning I caught the program terminating when it wasn't meant to.
I thought I saw a message go by, so I hooked up a video cam to watch the sequence. After running perfectly for 10 times, I finally caught the glitch.

http://www.ourcoolhouse.com/images/NXT_FileError.wmv

If you play the linked video, you'll see the program shut down at T=14 seconds. If you freeze frame it, you'll notice an interesting error message just as it dies. See attached pic

- File Error - Wassup ????

OK, I guess I have some more googling to do.

Note: I have checked my NXT memory and I'm only about 1/2 full.
57K out of 115K bytes used???

[emmell]

All:

Welcome to another episode of CSI: FTC. Excellent posts and great detective work.

Phil: Listening to the sounds of the movie you posted, it sounds as if you are manipulating your arm at the time of the "File Error". Maybe there is something that is causing a electric surge on the NXT cable from the motor to the controller to the NXT. Are all your cables okay? Any static issues possible? (I know that's an FRC issue this year). The observed behavior is that something is causing the NXT to abnormally terminate your teleop program. This kind of thing needs to get sent to PITSCO and HiTechnic (and maybe even NI) as this appears to be a problem with hardware causing the fault.

Mannie

[PhilBot]

Originally Posted by emmell

All:

Welcome to another episode of CSI: FTC. Excellent posts and great detective work.

Phil: Listening to the sounds of the movie you posted, it sounds as if you are manipulating your arm at the time of the "File Error". Maybe there is something that is causing a electric surge on the NXT cable from the motor to the controller to the NXT. Are all your cables okay? Any static issues possible? (I know that's an FRC issue this year). The observed behavior is that something is causing the NXT to abnormally terminate your teleop program. This kind of thing needs to get sent to PITSCO and HiTechnic (and maybe even NI) as this appears to be a problem with hardware causing the fault.

Mannie

Hi Mannie.

You've got a good ear. Yes in this instance it happened just when the arm was moving. It is possible that this is a common occurance, although it's hard to say because the fault is only really apparent when a motor screws up. The previous sound also indicates that the wheel drive had reached it's endpoint but had not been taken out of "servo" mode, so it sat there humming until the NXT timed it out. So it's likely that the actual problem had occured earlier, and it wasn't until the NXT commanded the next wheel movement (after the arm move) that the File Error occured.

Our wiring is pretty good (all tied in place, and protected from vibration etc) but I know that these motors can surge a lot. One problem with the very strict FTC rules is that there is no opportunity to use good electrical practices to minimize possible problems on a larger robot. For example, is it's required to daisy chain the power through all the controllers. The rules prevent using heavier guage wire, or making any soldered distribution bus to give each controller it's own dedicated power lines. I cringe at the current surge that must be going through the first set of screw terminals. I also wish that I could use a resettable fuse (like FRC) on the motors to prevent that horrible burned wire smell from being set free.

My problem is that the fault is random, and unlikely to be able to be replicated by Pitsco or NI. When doing my earlier encoder tests, I was able to generate some good plots showing the problems, and these were generally addressed by NI, but now they are less reproducable.

As you imply, I do think that the robot structure and code complexity is causing a problem.... just HOW is the question.

One problem with the NI code is that there are lots of places where there is an "error" output from a particular VI. The issue is: what does this mean, and what should we do about it. If a move command generates an error output... is there really any way to recover? probably not. I should probably just abort Auto and wait for teleop.

Both NI and Hitechnic have been helpfull, but both imply the problem lies with the other's equipment. No-one is eager to step up to be the systems integrator and really address the problem. It appears to be falling on the end-user to do this.

[PhilBot]

Two ideas gelled for me today, and made me go Hmmmmm.

After a bunch of perfect test auto runs, I had my File Error occur on two runs in a row. It occured to me that I hadn't powered down the NXT for a while, so I did, and things started working again.

It's amazing how we assume that computers "just run" these days, because it took me this long to suddenly think "memory leak".

All of a sudden I wondered if there is a memory leak in the labview NXT firmware, and perhaps these problems occur based on how long the NXT has been running, or how many times a certain type of operation is performed.

I did some more research on the "File Error" message and it seems like it may be related to 1) An invalid array index or 2) failure to allocate dynamic memory.

I could see how a memory leak could cause #2.

I also wonder if just sitting on the playing field, running the code template for 2-3 minutes could cause a memory leak to grow to the point where it shuts down the NXT program when you start really doing stuff.

On a related note, during FLL season, I casually used an FTC NXT in one of our FLL robots (that was programmed in NXT-G). This NXT had been upgraded to firmware version 1.12. After several instances of the NXT program just locking up while it was running, I went back to the "release" 1.05 firmware. Never had the lockups again.... Hmmmmm.

Does anyone know if there is a way to read the amount of "free memory" while running a program on the NXT.... it might be enlightening...

Phil.

[emmell]

Hmmmm.... I wonder if ROBOTC users see this as well.

NI/PITSCO - please notice this and investigate.

[ssi]

Phil,

I can't help you with checking available memory while running, but we have seen file errors before. However, all of these could be traced back to coding errors. At the moment, I cannot remember exactly the problem, but I thought it had to do with attempting to talk to a motor or servo controller on the wrong port (or wrong place in the daisy chain). At the time we were probably using NXT-G (but could have been LabView, we went through all 3 packages before settling on RobotC). Is it possible that you have a coding problem in a section of code that is only run on rare occasions possibly due to the position of the robot at the end of autonomous/beginning of Teleop.

As for the frustration of the robot not responing to controls, our team managed to experience that twice at the MD competition. The first time it was caused by a team member forgetting to plug in the external battery in when we swapped for a fresh one. Fortunately our team mate won that qualifying match for us. The second time was in the semi-finals when the RobotC software was not detecting the external battery and refused to run. After the match, we unplugged and re-plugged the battery and it registered normal!?!

We competed 1 week later at the PA tournament and had no control or connection problems with our robot (we also reset bluetooth between all matches). However in our teams second semi-final match at least 3 robots locked up (though one may have been broken) just as our team mate was about to score 2 racks. This would almost have to been an FMS or USB problem.

David

PS. Good luck at Atlanta

[ptan]

Testing on RobotC:

So far I have not yet been able to recreate Phil's FMS problem, nor have I seen the File Error problem to date. We are using RobotC, so that may be a factor in the system. I have experienced bluetooth problems when the NXTs auto power down.

I will keep on testing to see if I can recreate the problem.

Paul Tan.

[basicxman]

Originally Posted by emmell

Hmmmm.... I wonder if ROBOTC users see this as well.

NI/PITSCO - please notice this and investigate.

I have done lots of testing with RobotC on two NXT's, I haven't had any problems yet. Although at the competition I had this happen to me, my alliance partners and opposing alliances many many times. This needs to be solved. Hopefully before the championship as it's probably the most backlogged tournament already.

[ptan]

I've barely been able to recreate what actually happened to us at the tournament -- basically at the Ontario FTC Championship, we saw our robot keep moving, even though we had lost all joystick control.

We've been at this for the past week, so this is the first time that we actually saw it happen again.

On the screen of the NXT, it shows that the bluetooth was disconnected (the diamond beside the bluetooth symbol became a less than sign), but the teleop program was still running. The motors were still doing the last thing we told it to do before the bluetooth connection got lost -- AND THE ROBOT DID NOT SHUT DOWN THE MOTORS after the connection was lost.
The NXT screen was still showing "Teleop RUNNING"

I don't know if we can recreate this problem again or not as it was the first time we saw it in the past week of testing and practicing. The FMS showed that the robot was still connected on bluetooth.

This is a brand new NXT (out of an unopened box as we had replaced the original NXT thinking it might have been flaky), Robot C 1.46 with the latest firmware, running the competition templates.

[dickswan]

Originally Posted by emmell

Hmmmm.... I wonder if ROBOTC users see this as well.

NI/PITSCO - please notice this and investigate.

The "File Error" message is a general catch all message and could include memory leaks or other things that cause an abnormal user program exit.

ROBOTC does not use a single error code and has about 50 unique exception messages that it checks for.

Just thinking about this, in NXT-G, displaying text on the LCD screen performs memory allocation and free for the strings involved. So depending on how frequently the screen is updated, there could be a lot of memory allocation for a robot sitting idle.