Robots not under driver control- does it happen- do you determine why?
 

I'm curious about the prevalence and root cause analysis of issues controlling robots during competition:

Has your drive team ever lost complete control of the robot for a second or more in any match?

Has responsiveness been delayed by more than a quarter of a second and made it unnatural to drive?

Did you have the proper analysis tools to find out a root cause to the problems?

What percentage of control problems were you able to isolate?

What percentage of problems didn't show up again, but had no clear indication of what caused the problem in the first place.

My perception is solid problems are isolated and solved clearly. Intermittent problems and sluggishness are sometimes considered part of life instead of being isolated and resolved.

How do best teams determine that they have enough safety margin on their radio signal to never worry about being in a hard pushing scrum with three other robots in the corner farthest from radio access point?

We were brainstorming about logging voltage, radio signal, transmission delay to the DS and transmission delay from the DS at a 100ms rate and link it to video for the matches.

By the way, the DAP-1522 manual:
"Keep your product away (at least 3-6 feet or 1-2 meters) from electrical devices or appliances that generate RF noise."
ftp://ftp.dlink.com/Multimedia/dap15...manual_120.zip

Re: Robots not under driver control- does it happen- do you determine why?
 

I know we've had responsiveness problems in the pits. They were always associated with a ton of CAN errors. I've yet to discover the root cause of the CAN errors, aside from one of the jaguars always flashes. I'm certain the fact of the CAN errors is what causes the delayed responses, since the CAN libray uses blocking calls, which means everything will stop until the command times out or gets a response. So the program hits a snag, then eventually times out and updates everything with rather older joystick values that haven't been refreshed.

Re: Robots not under driver control- does it happen- do you determine why?
 

Yes, we've had this intermittently, but it never was a major problem, I believe. Last year with the new Classmate there was an internal issue that would periodically trip the watchdogs at a concerning rate, but we fixed it after searching on Chief Delphi. We sometimes get errors in the diagnostics tab when starting up, but we are quite confident that this is a common occurrence during boot-up (it doesn't affect control since we haven't enabled yet). Most of the time when the robot hiccups it's because of a watchdog error, although most of those times we don't even notice it; it seems to be instant. We often can't think of what might be wrong, though. But we would check for lagging code, the diagnostics tab of course, a dead battery, if the drivers are cranking on the controls and drawing too much current...mostly basic things (we don't have a plan like you mentioned). Sometimes we do image the cRIO and that's usually a good "reset" point to start from.

Re: Robots not under driver control- does it happen- do you determine why?
 

Always flashes as in the same flash as a disabled Jaguar? (Slow yellow flash?)
Or an intermittent blink? Solid yellow (or Red/Green) - off - working again?

399 had the second during build. It manifested itself as a lag, especially when driving backwards.
We were able to narrow the cause to a single Jag [the one that intermittently would flash], and by removing it solved the unresponsiveness and all of our CAN issues.

Under BDC-COMM, that Black Jaguar would remove itself from the CAN bus and not respond until it was re-enumerated, when it was commanded to go a certain direction [Backwards].

Re: Robots not under driver control- does it happen- do you determine why?
 

You know, I'm not certain if it's always the same jag. It was doing the slow yellow disabled blink and we couldn't see it on the CAN bus, until we cycled power on the robot and everything was fine. I'll tell our programmer to note which jag is causing the problems the next time it happens and we might just swap it out for a spare one if it's the same one.

Related note: was your problem jag a black jaguar on your drivetrain or another system subject to sudden heavy current reversals? The gray jags have issues with sudden reversals blowing up gate drivers and disabling half the h bridge. Perhaps the new ones are subject to some other, more unlikely sort of failure when hit with a sudden reversal.

Re: Robots not under driver control- does it happen- do you determine why?
 

You can sent the Jaguars to limit the rate of voltage change over time.

/**
* Set the maximum voltage change rate.
*
* When in PercentVbus or Voltage output mode, the rate at which the voltage changes can
* be limited to reduce current spikes. Set this to 0.0 to disable rate limiting.
*
* @param rampRate The maximum rate of voltage change in Percent Voltage mode in V/s.
*/
void CANJaguar::SetVoltageRampRate(double rampRate)

Team 241 needed to add it last year to keep from blowing the original Jaguars.

I've read that some teams reinitialize their Jaguars controlled by the CAN if they become nonresponsive.

I think if CAN timesout during initialization, the Jags may misinterpret or ignore the commands you typically send during periodic code time.

Re: Robots not under driver control- does it happen- do you determine why?
 

It used to be on the drivetrain.
We noticed that it sometimes stuttered (and would produce a whole bunch of CAN errors) and then we took it out.
When it was run under BDC-COMM, it wasn't connected to any motors just the Serial port and terminator.

Re: Robots not under driver control- does it happen- do you determine why?
 

We had a problem with our robot, where it would randomly lose control during matches, but not in the pits. It also happened on the practice field.

We systematically removed things, even the bridge, so we knew it wasn't that. It turned out that it was the camera, making the C-Rio process too much.

Re: Robots not under driver control- does it happen- do you determine why?
 

We figured out it was the camera as well. I had noticed during development that on occasion the camera image being displayed was 3-5 seconds behind live. So some fairly major buffering is occurring from the cRIO. If the bandwidth is marginal during a match(all teams using cameras or simply interference) then it is reasonable that any commands and responses get stuck in a queue behind video that is being buffered.

We are using Java and could never get the camera commands to set lower resolution/quality. With version 28 our camera connection stopped working for analytics in autonomous mode.

Trading email with wpi I got some advice to make sure the M1011 firmware was updated and to put the camera in DHCP mode versus static IP as indicated in the install guide. This implies the cRIO has a DHCP server and using axis software library to discover cameras. Made the changes and sure enough camera started working again.

Now the bad news. During practice at regionals they had a ton of problems getting robots to register. They fixed the problem at 3:30 and we were able to get in some practice sessions. Autonomous mode in our code was a big problem and we would loose the robot for short or long periods. Since we couldn't find the problem we turned off Autonomous mode but left the camera init in the code.

Our first three matches we had a dead non-controllable robot. Finally figured out issues with bandwidths could be the problem and then occurred to me that the camera was still streaming video and that wasn't helping. Disabled camera in the code and for the most part our robot control problems went away.

We finished 26 out of 60 at Florida Regionals as a rookie team where we really wanted those three matches back and would have placed much higher. We have a very reliable minibot, can score the top row and had autonomous mode working until it was disabled and couldn't spend time with all the other problems with communications. Just found out we won the Rookie Allstar award so we get to go to nationals and will see if we can get another shot at being a competitive robot.

Need to figure out in the Java api if it is possible to use the camera in autonomous mode and then disable it during the match. Has anyone had any luck setting the camera resolution/image quality or disabling the camera from streaming after startup?

Re: Robots not under driver control- does it happen- do you determine why?
 

i lost complete control of my robot for a entire match for a unkown reason while competing myself. my robot would not respond at all. the people behind the desks said that "everything on their end was fine, so it must be us". but when we went back to the pits and tested it, it worked fine. this happend to a few other teams throughout the day. i was getting very annoyed with the whole thing myself.

Re: Robots not under driver control- does it happen- do you determine why?
 

This is exactly what happened to us with CAN on jaguars at Traverse City. We sit still a match. Go back to test it at the pit and everything looks fine. Go out for the next match and the robot sits still again. All this time it says communication is fine. Decide to control our drive instead by PWM and CAN stopped pestering us until a jaguar blew knocking out our entire robot because the CAN stopped the robot due to inability to find all jaguars.

Re: Robots not under driver control- does it happen- do you determine why?
 

What language were you using? 2CAN or Serial?

Re: Robots not under driver control- does it happen- do you determine why?
 

We were using the 2CAN and Java.

Re: Robots not under driver control- does it happen- do you determine why?
 

Was everyone with communication problems streaming video?

Re: Robots not under driver control- does it happen- do you determine why?
 

Our team encountered no robot control twice this weekend. During both matches the autonomous worked, but once the robot switched to teleop there was no control at all. The team was told both times that the field showed the robot was communicating and the driver's station showed no loss of comm.

I quizzed an alumni who was volunteering for a little more information and he stated that they found that our driver's station was showing no signals from the joysticks. However, after discussing this with our driver, he had diagnosed the joysticks while the match was running and found that the driver's station was seeing them correctly. Keep in mind, the field and the driver's station showed no errors what-so-ever.

After this match we took the robot back to the pits WITHOUT TURNING IT OFF, and tethered it to the driver's station. The robot continued to not respond to any teleop control, but showed that it was communicating and the cRIO was flashing the User1 indicator, which generally indicates that teleop is active. We rebooted the program and it worked fine after that point.

The frustrating bit is not once could we duplicate this while tethered and it never happened while testing the robot for two weeks during the build season. So far it has only happened when connected to the game field. Unfortunately, one time had to occur during a quarter final match. :\

It has been very disappointing and I hope something is done to address it soon as we haven't been able to determine any obvious cause.


Edit: I should point out that our robot is controlled using Jaguars only. Four Jaguars run off PWM signal to control the 4 drive motors (mecanum). Another 4 are operated off CAN to control the tube mechanism. None of the Jaguars work at all when this occurs.

Re: Robots not under driver control- does it happen- do you determine why?
 

I did find in testing that once you enter in autonomous mode you need to keep track of the time spent and return from the method after 15 seconds. I have a keepGoing() method that I put in all the loops and before the next step that forces a return if 15 seconds has expired.

If not, the joysticks are not sampled and the appropriate drive method called. We are doing Java but assume the program constructs are the same in labview or C++.

Just because the field switches from autonomous mode to telop mode does not impact the code you have running on your robot.

Re: Robots not under driver control- does it happen- do you determine why?
 

We had talked about the possibility of the robot not going out of autonomous mode, but wouldn't that cause the User1 indicator to turn off? I'll admit that this assumption is not based off direct knowledge, but what I had been told by the mentor who is experienced with LabView and the cRIO.

Re: Robots not under driver control- does it happen- do you determine why?
 

Stuck with PWM output this year (absolutely no reason to switch to CAN -- everything that can be done with it can also be done by coding the Crio) and lost our camera. Everything runs so much smoother now.

Re: Robots not under driver control- does it happen- do you determine why?
 

Unless lab view does something special to understand the notion of autonomous vs telop I think it will simply do in a program loop or block of code what it is asked to do. If you spend 30 minutes in a loop adding up numbers no reason why the code would return/yield/interrupt that block of code because the mode changed on the field.

Re: Robots not under driver control- does it happen- do you determine why?
 

The way the Labview framework is written you cannot get stuck in Autonomous mode. When the field changes over, your autonomous code is terminated whether you are in a loop or not. If you change any of the framework or put your auto code somewhere other than the Autonomous Independent VI, all bets are off. Also I'm not sure if this functionality will work or not if you are in a loop that never yields the processor (IE no loop delay).

This is not the same as the C++ and Java frameworks which have no such functionality (from what I've been told)

Re: Robots not under driver control- does it happen- do you determine why?
 

The only reason you should not be 100% all of the time is code or electronics, it's as simple as that. The only time we had problems with controls is when we had bad electronics, loose wires, or a discrepancy in the code

Re: Robots not under driver control- does it happen- do you determine why?
 

We are using serial CAN with all Black Jaguars (programmed in Java) and had no control problems (such as lag or lack of control) at the WPI Regional. There only instance our robot was "not responding" was when our drivers forgot to shift into low gear in a pushing match and tripped the Jaguars current protection for three seconds, but they only made that mistake once.

This may or may not be related to our stability with CAN, but early in the build season our programmers noticed a conflict between the camera and CAN code in Java, and they fixed the bug.

Re: Robots not under driver control- does it happen- do you determine why?
 

I agree the inconsistent CAN errors (such as ours) that people have gotten are a function of code. The problem is that they're near-impossible to reproduce in a controlled setting and its absolutely impossible to diagnose where the problem is occurring.

I highly suggest switching to PWMs if you're having this issue. The wiring sucks, but its worth it for consistent control if the robot.

Re: Robots not under driver control- does it happen- do you determine why?
 

The theoretical maximum transfer speed over wireless is 5 MB/s. The actual maximum is less, due to
packet and header overhead. The camera can easily saturate the link at that speed. When congestion
starts to drop packets, it makes the problem worse over tcp, as the lost packets have to be retransmitted.

1000 mbit / s = 100 MB / s
_100 mbit / s = _10 MB / s
__50 mbit / s = __5 MB / s

We did some experiments with the camera and the crio.

Ex.0 (wpi lib) receive images via http on crio and retransmit to DS via TCP.

This failed last year (2010) when we tried to run video. FIRST restricted all video to run over TCP.

Positives:
- ???
Negatives:
- data sent over TCP, lost packets are re-transmitted
- camera data saturates network link disabling robot communications
- crio tcp network stack backs up setting off hardware watchdog
- did not reach 20 fps, camera task never yields

Ex.1 receive images via http on crio and retransmit to DS via UDP.

This was our first idea this year (2011) to improve the situation. We got approximately 4 img/s.
One drawback to this experiment is that the crio task for receiving the images from the camera
and sending them to the DS is never idle, so it takes away available CPU cycles from controlling
the robot. We are not currently able to measure what performance other tasks impact the processor.

Positives: data sent over UDP, lost packets are not re-transmitted
Negatives: did not reach 20 fps, camera task never yields

Ex.1 receive images via http on the DS and display them at 20 fps.

Positives:
- 18 to 20 fps display on DS
- packets backup into camera not crio
- no hardware watchdog timeouts on crio
Negatives:
- data sent over TCP, lost packets are re-transmitted

Re: Robots not under driver control- does it happen- do you determine why?
 

Our flaw was overused, overheated motors without the proper gear ratio and not enough motors (we only used 2 CIMs).

Re: Robots not under driver control- does it happen- do you determine why?
 

We lost control a few times at FLR -- the robot would sit still with no coms for about one minute, and then kick back on. We're 99.99% positive it is the bridge rebooting, but we have yet to figure out why that is the case...

...It's got good location on the robot (up high, not surrounded by too much metal), and we double-triple-quadruple-quintuple-and-more checked the electrical connections.

It was intermittetent, frustrating, and cost us a match three times. :/

Re: Robots not under driver control- does it happen- do you determine why?
 

We were in New Youk over the weekend and had 2 instances of comm problems. We are using C++, no CAN, 4 victors and 1 jaguar all PWM controlled, no camera.

The first time it happened in prelims. We were about 1:30 into the game and we were lining up to deploy but we just stopped. FTA said is was us. No power issues, no wire issues.

We tie wrapped the ethernet cable in to the bridge.

In Quarterfinals. we started autonomous. In our design the tower is pushed vertical by pistons. They started up then started down, then up, then down. The only way to get the towers down is an input from driver or operator (button 5 ). They were not near controls. Autonomous for the fisrt part goes forward for 4 seconds. We did not go as far as we always do. Looked to me like the CRIO was restarting the code.

Once in round we were good but again with about 30 sconds left we died. This time I was less than pleased. I made sure I got the FTA's attention. He came back to the pit and looked all over and found nothing.

My only issues may be the placement. We are moving the radio more to the open.

Another team that was in the semifinals had a similar experience.

Very frustrating...

Re: Robots not under driver control- does it happen- do you determine why?
 

At a pre-ship scrimmage, our robot would occasionally start spinning in circles or take off at high speed in reverse. This would happen with nobody touching the driver joysticks. The program was definitely using only those joysticks to control the drive motors. Some of the team believed it was a communication issue, but I couldn't see any evidence of that kind of problem.

It took us most of the day to track down a bug in our rate limiting code. It was intended to keep the robot from accelerating backwards quickly so a chain tensioner wouldn't be overstressed. The bug was being triggering whenever a joystick went just barely outside the deadband in reverse. We spent a few minutes puzzling over the code, first trying to figure out how it was broken, then trying to figure out how we had intended for it to work in the first place. Finally we just threw it out and rewrote what it should have been originally.

Re: Robots not under driver control- does it happen- do you determine why?
 

We also have pneumatics for lifting our fork lift and have seen that behavior multiple times where it simply retracts the forklift. Something is causing the code to go into the startup state. This has happened to us in practice in the shop so the problem could still be communication related where a drop in communication does the same thing as not feeding the watchdog.

Not sure if it is possible to monitor the communication from the cRIO to have it log anytime it goes into "safe" mode.

Re: Robots not under driver control- does it happen- do you determine why?
 

If you are at an event get with the FTAA before your match and ask him to pay attention to your robot. He has a display and logs of the communication status between the field, the robot, and the driver station. He can also spot a big drop in battery voltage, which indicates other problems. The logs can be checked but it is easier to just catch stuff as it happens most of the time.

Re: Robots not under driver control- does it happen- do you determine why?
 

I could not find this information in any FIRST user guide I'm aware of.
I would guess that the best/most experienced teams already know to do this. Unofficial information flow can sometimes unintentionally give an advantage to those in the know versus those trying to rely on official channels and not be a burden by asking for types of help.

A typical relatively new team sees the FTA as extremely busy with general responsibilities and focused on getting connections from all 6 DS to all 6 robots at the beginning of a match. The FTAs have to choose where to put their focus and if every team that had occasional control problems asked the FTAs to monitor their robot in every match, I think the FTAs would be overwhelmed: (this non-statistically accurate polling points in that direction).

For example, at GSR, when CAN timeouts seemed to percolate up as an issue: teams known to have CAN were informally polled by teams or by the FTAs: not all teams that had CANs were advised of the seemingly common issue in a timely manner- some played through Saturday with CAN while many teams (especially the better informed teams) had switched over to PWM by Friday
Obviously there is a balance of discretion about conclusions, but statistical observations by FTAs should probably be shared with all the teams at the same time, not just all those that ask or are in the know. It's also very clear to me that the FTAs are always trying to be as fair and generous as possible. Now the CAN timeout issue was probably a special case that does not happen very often. Working with the FTAs, the experienced teams form their own informal networks of FMS field issue information.
If several teams make a decision not to use a feature based on that collected information, it would be nice if the FTA broadcasts it in a timely manner.


Can someone post a picture of the real time log an FTA has available to them ?

Does FMS take requests for enhancements? How about logging all the detailed data per unit time and storing by team and match number?
Then each team could request only their data at any point during the competition. Also, with all this data logged, enterprising FTAs could look for issues and publish general results on problems teams are having.

Does the FMS logging have signatures for each of these problems that can be shared with everyone?

1) Radio placement
2) Radio proper cord and strain relief
3) Charged battery
4) shorting motors
5) software errors that cause UserProgram to run too slow (all the time)
6) software errors that cause UserProgram to run too slow (occasionally)

Can the same logging that FMS provides be made available in the driver station or the FMS light code?

Many teams work hard all season for only about 25 minutes of total actual playing time. (Some even less if their robot doesn't make the practice matches on Thursday).

I'm on a mission to reduce control problems for everyone especially the nubies.

A more rigorous (and scientifically accurate) poll at competitions to size the problem is probably the first step.

Re: Robots not under driver control- does it happen- do you determine why?
 

Having pneumatics in the on state and getting a momentary glich that turns it off causes a very dramatic movement that is visible. If a motor is spinning and you get a momentary toggling of output you won't see it.

Since more than one group seem to have this problem we can probably eliminate or zero in on common elements to find the source.

The problem appears to occur in both C++ and Java so not a language issue.

Camera and bandwidth issues

Wireless communication

Optical encoders generating interrupts that have a higher priority that need to be serviced and the watchdog timer doesn't get serviced. With two encoders and two inputs each 4 x 720/780 = 3000 interrupts a second. I am using 1X on the encoders which I think is serviced by the processor vs 4X which gets serviced by the FPGA and based on other forum discussion doesn't work with more than one encoder. I am under the assumption that the encoders are serviced as interrupts by software in 1X. The encoders give 720 points of resolution but when I rotate our wheel 10 times by hand I get very close to 7800. I have performed this test multiple times on two different wheels and aways get 7800+/-small error. If the resolution per turn was 720 then I would expect 7200+/-small error. I can work with the 780 but just indicates something isn't working as expected.

In looking through the JavaDoc more than one method to set watchdog and safety related attributes. Does anyone have any details regarding the need to set these parameters or does the default template work assuming you feed the watchdog?

Re: Robots not under driver control- does it happen- do you determine why?
 

I noticed last year when we were using LabView that running the program from LabView and not as a start-up program significantly slowed down response and even caused some dropouts. I think this was just because the program was running on the ClassMate, so any time it slowed down (frequently!), the code slowed down. Also in LabView, the camera seemed to both be very slow and slow down the rest of the code, to the point of dropouts and even errors sometimes. This year we switched to Java and all those issues went away (mainly because the code is always a "start-up" program, it is run on the cRIO independent from the computer).

Re: Robots not under driver control- does it happen- do you determine why?
 

Encoders do not interrupt the cRIO. They are always handled by the FPGA. 4x encoders use dedicated quadrature circuitry, while 1x and 2x use dedicated counter circuitry.

Where did you get encoders with 720 counts per revolution? The ones in the Kit of Parts this year are 360 cpr, and 250 cpr encoders are common. Is the wheel on the same shaft the encoder is on, or is there a chain and sprocket? Has the "distance per count" for the encoder been set in software, or is it left at its default value?

Re: Robots not under driver control- does it happen- do you determine why?
 

Thanks for clarifying the interrupt issue as currently focused on figuring out the non-responsive robot problem. I haven't had a chance to look through the cRIO docs to see what is under the hood.

The encoder is on the shaft of the gear box provided in the KOP. So a single wheel turn is probably giving a 2X increase on count. I initially went with 360 as one revolution of encoder and 720 for one revolution of the wheel. I have two encoders one for each wheel. I set the distance per pulse and in distance traveled simply didn't match distance measured. Put the robot up on blocks and turned wheel 10 times and get raw pulse count. Both wheels in the test indicate 780 pulses per wheel revolution.

We ended up building a second identical robot and ordered two additional optical encoders from andymark. Same exact results in that it appears to be 780 pulses per wheel revolution.

In theory the gearing ratio could be 2.166667 from the wheel to the rotation of the optical encoder shaft and the reason for the non integer multiplier. Always feel better when you have nice integer multipliers.

Re: Robots not under driver control- does it happen- do you determine why?
 

We're still trying to figure out why we're getting COM drops.

Our wiring is sound (and connections both strain gauged and either hot glued or taped).
Our power appears to be constant.
Our bridge was replaced with another bridge.
Our bridge is up high and visible everywhere but the bottom (it is upside-down) from 360 degrees with very little obstruction from the aluminum frame.
We are not getting errors as we run in autonomous or teleop.

...and yet, a few times at FLR, and a few times during practice, our bridge likes to reboot, killing our robot for about a minute.

Suggestions, anyone?

Re: Robots not under driver control- does it happen- do you determine why?
 

This is a simple live display that the FTAs/FTAAs/CSAs use.
It's actually pointed at the field and can be video taped if you want, as long as no referee happens to be blocking it.

The logged data is much more intense and the FTA generally just checks it for obvious issues, such as low battery voltage and communications delays and drops. It is by team/match/etc. as you'd expect. Sorting through the multiple potential causes is more what the CSAs back in the pits with the teams do to relieve the strain on the FTA.
Generally, if an FTA sees a team having a problem on the field, he/she will ask a CSA to follow the team back to the pits to help troubleshoot.
Ultimately, it's vastly easier to learn to troubleshoot the problem from a volunteer who's already seen similar problems on 60 different robots.

The time to review the recorded match data is limited, however, because of the ongoing matches. It's much easier to ask the field crew to monitor it for you as your match is happening.
It doesn't have to be the FTA that watches it for you. Regionals have a help desk of some sort. I worked the NJ Regional, for instance, and we had about seven people available (counting FTAs), any one of whom could monitor the feedback for obvious issues.

You can also log your own data on either the cRIO or Driver Station.
Battery voltage, packet timing, etc. are easily collected.

Re: Robots not under driver control- does it happen- do you determine why?
 

Just the obvious.

If your radio drops out for more than a minute, before it comes back, then it's not a position problem, it's a power supply problem.

• Watch the low point of battery voltage drops while the robot is under strain. Consider adding an error message to the Driver Station whenever the battery voltage drops below 6v or so.
• Verify the radio is on the special power protected PD output, of course. Found way too many on the camera 5v feed or regulat red/black wago connectors.
• Wiggle the barrel connector while under power in the pits to see if you can force a power outage. That's how we identify poorly fitting connectors (Radio Shack :-) ).
• Inadvertent contact with the button on the side of the radio.

Re: Robots not under driver control- does it happen- do you determine why?
 

What power connector are you using? There seems to be a hint that the DAP's power pin might be a tad smaller than an apparently correct connector is designed for. If you aren't using the connector that came with the DAP, it might be losing contact due to vibration of the board inside the case.

Re: Robots not under driver control- does it happen- do you determine why?
 

Mike Betts posted something very interesting that might be related to your problems: Is it the right plug?

http://www.chiefdelphi.com/forums/sh...ad.php?t=93657

Re: Robots not under driver control- does it happen- do you determine why?
 

Mark,

How do we get the packet timing? We are using C++.

-Hugh

Re: Robots not under driver control- does it happen- do you determine why?
 

Thanks for the suggestions. Here's what I've got so far:

1. Our voltage certainly is not dropping below 6V. We've monitored it closely just to make sure.
2. We are using a WAGO connector to plug the 5V inverter into the dedicated, protected PD 12V output. The right-angle power connector that came with the radio is connected to the radio and taped in with electrical tape wrapped around the entire radio to keep it snug.
3. Attempts to force a power outage by wiggling any of the connections have not worked.
4. Our radio is set up such that the switch will not be hit by anything, and we are 99.9999% certain that it was not hit by anything at any point where we lost communication.

We're going to re-wire everything anyway and test it on our practice bot before Championship. Hopefully whatever the issue is, that solves it.

Thanks for the help, folks!

Re: Robots not under driver control- does it happen- do you determine why?
 

Perhaps your 360 CPR 5v converter is flaky. Try banging on it and see if you can prompt a power outage that way.

Re: Robots not under driver control- does it happen- do you determine why?
 

Good call. I'll try that!

Re: Robots not under driver control- does it happen- do you determine why?
 

The easiest most efficent way is to just take the summary from the DLink statistics page. At least that's what I prefer to use since the DLink is made to collect that kind of data. It's nice because it's also the point that divides the transmission statistics into wireless (DLink<->field router<->DS) and Ethernet (DLink<->cRIO). We actually get additional results that the FMS logger doesn't have. In the past I've been able to identify a specific bad pin on the cable between the bridge and the cRIO this way.

Either captured by a query script (maybe while disabled after Teleop is over), or just examined via a web browser after a match. To examine it after a match be sure to leave the robot powered to preserve the match data until you can hookup your laptop.
Of course, if your DLink is rebooting then the data does get lost, but at least you'll have proof that it's rebooting.

I'd hesitate to log too much data too quickly on the cRIO, because of the potential for overload
If you're running a more powerful laptop as the driver station, logging the majority of the data back there might have less system impact, as well as being more easily accessable.

Re: Robots not under driver control- does it happen- do you determine why?
 

Do you have an existing query script you would be willing to share that we could work from for accessing the DLink?

Is there a description of the various modes of WiFi that the FRC field will use?
(E.g., 2.4G versus 5G, 802abgn)
Is hunting between modes ever supposed to occur?
Does the FTA check to see that team have provisioned other modes consistently other that bridge/AP?

The document How_to_Configure_Your_Radio_Rev_A on page 11 seems to imply 2.4/5G, 802abgn, no autoscan, channel 6, channel width 20MHz is standard competition configuration.

E.g., do all team need to be using auto chan scan or are teams allowed to provision their DLink to camp on the unshared channels?

Re: Robots not under driver control- does it happen- do you determine why?
 

A comment on the difference between Run as Startup and pressing the run button using LV.

Running LV with only the run button is quite similar to running as startup, but because the panel(s) are open, the cRIO and development computer are both working a bit harder communicating info back and forth about panel values.

This protocol is typically lightweight, just sending binary values, usually from robot to dev computer. If you change a panel value such as a PID gain, the dev sends a small packet to the robot. Obviously, the more windows you have open, the more traffic. The more probes you add to your VI, the more traffic. Vision is a case that is not lightweight. It is extremely useful to be able to probe and view panels with camera data on them, but to get the data to the laptop, the image has to be compressed by the cRIO and transmitted as part of the protocol. This only happens if the image is visible on a probe or panel. Scrolling the image offscreen won't help, and I doubt that minimizing helps. To lighten the load, you close the panel with images when you aren't using them.

I develop with the run button all the time, but then again, I like debuggers. Also, I usually don't run the dev stuff and DS on the same computer, and almost never run both on a Classmate. Hopefully this helps explain why panels can indeed cause dropouts, but it doesn't have to force the Startup approach too early.

Greg McKaskle

Re: Robots not under driver control- does it happen- do you determine why?
 

Our team has had some problems.
Our 2009 robot was always a little sluggish - we know it to be a bug in the programming.
Our 2010 robot usually worked well. There was one match where our robot began behaving eratically without control of the drivers. The entire alliance did, so we believe it was a field fault.
Our 2011 robot has not had any problems yet.

We usually place our radio somewhere exterior on the robot, but protected by a sheet of Polycarbonate. Our 2011 robot has it completely unshielded (it is in a position where it would be very difficult to be hit) but our 2010 robot had a piece of 3/8in Polycarbonate armor (yes, that's bulletproof).

Re: Robots not under driver control- does it happen- do you determine why?
 

The radio gets reset and reprogrammed at the Radio Kiosk automatically. There's nothing for the FTA to check, unless a team went in and changed something after the radio programming.

Re: Robots not under driver control- does it happen- do you determine why?
 

So if the radio is "factory reset" to defaults and then specific parameters get set at the Radio Kiosk.
Which parameters end up being different than the factory default, besides Bridge mode and IP address and mask and gateway?

http://www.usfirst.org/roboticsprogr...t.aspx?id=8632

The FMS Delta software will be pre-installed on a laptop and shipped with the playing field electronics. FIRST will also ship a lower cost field access point, a Linksys WRT610N or similar, configured for 802.11N @ 5GHz.

I'm trying to figure out if 802.11N @ 5GHz is the only mode that a field access point will ever use. If it is, then why does the competition settings of the DLINK DAP-1522 not specify these particular options?

The shaded picture in the document How_to_Configure_Your_Radio_Rev_A on page 11 seems to imply 2.4/5G, 802abgn, no autoscan, channel 6, channel width 20MHz is standard competition configuration.


I think we had been using 2.4G in our practice: it might give us better range for practice, but won't uncover weak signal issues we might have when using the lower range 5GHz in real competition.

Re: Robots not under driver control- does it happen- do you determine why?
 

The output shaft of the gearbox has a 12 tooth sprocket, the wheels have 26 tooth sprockets: 360 (encoder counts) * (26/12) = 780

Note: An encoder labeled "360" will give 360 counts at 1X as it counts only the leading edge of A, 720 using 2X, counting leading edges of A&B (or leading and trailing edges of A), and 1440 using 4X which counts leading and trailing edges of both A&B signals.

PS. The cRIO FPGA handles the signals from the encoders and does not cause interrupt overload. Some teams have the encoders wired into CAN controlled jaguars - these do not cause interrupts either.

Re: Robots not under driver control- does it happen- do you determine why?
 

Losing control has never happened to us when not on the field, but when we were on the field we lost control twice, and one of those two times the e-stop would not register either, and our robot kept running after endgame.

The second time, the e-stop did trigger.

Re: Robots not under driver control- does it happen- do you determine why?
 

In our first match at Seattle Cascade (which was qualifcation match 1), we had no control of our robot. about 30 seconds into the match we realized that we had forgotten to plug in our joysticks :ahh:. after we plugged them in, they didn't connect till after the match. the next couple matches we had issues of burning out jaguars, which with mecanum wheels causes driver control. after replacing a few and remounting them to prevent overheating, we were fine the rest of the regional.

Re: Robots not under driver control- does it happen- do you determine why?
 

Once the match starts, the DS no longer enumerates looking for changes to joystick connections. It just takes too long. If you change joysticks in a match, hit F1. It will force an enumeration and you should be good to go.

Greg McKaskle

Re: Robots not under driver control- does it happen- do you determine why?
 

We had similar problems as team 935 during the Sacramento Regional. The robot seemed to work for one round, then we would come back to the pits, and in the next round, autonomous worked but teleop did not. We would redeploy the code and it would exhibit the same pattern. It would work once and then next time it wouldn't work. At one point the controls, when working, were sluggish.

We were not able to get field time to diagnose the problem on the field. It worked fine tethered in the pits or on the practice field. Finally the field operators collected some data that showed that we were lagging in field communications compared to the other teams. In the pits we were allowed to use the old radio to try and diagnose the problem. We were able to show that there was not a lag with this set-up.

We use PWM for our motors, no CAN, and no camera (although there was still some initialization code in the begin.vi).

We are still desperate to fix this problem before San Jose Regionals on 3/31. Any insight would be appreciated.

Re: Robots not under driver control- does it happen- do you determine why?
 

We ultimately found a number of problems that could have effected our robot. Going down the list:

1) The classmate PC had a broken ethernet port that prevented the cable from clicking into place. If memory serves, the cable only needs two specific wires to be connected for it to think there is a proper connection (even though some data wires may not be connected). This may explain our dead robot during teleop while the field showed we were connected.

2) In labview there is no error handling in the CAN-Jag VIs. The potential is that if there is a CAN error and the cRIO does not at least accept the error (it doesn't necessarily have to do anything about it), it could cause the cRIO to freeze or lead to partial functionality of the CAN network.

3) During autonomous it was evident that the robot was not being fully killed when autonomous ended. At the end of one of our autonomous routines the robot is commanded to rotate at the very end of the period. Often times this command is chopped by the end of the period. After reviewing replays we found that on some occasions the robot would stop rotating once the period ended, then after a brief pause would finish the command as teleop was starting. This, however, did not coincide with our occasional loss of comm or occasional loss of some of the CAN-Jag controllers.

Ultimately we decided to not roll the dice and took a number of precautionary measures in the week hiatus between Kansas City and Oklahoma:

The autonomous was no longer commanded to rotate at the end of the period.

We rewired all but one of our Jaguars to PWM control. The one remaining Jag was now technically operating off RS-232. This one was necessary as it used an encoder to determine arm position and we did not want to reprogram the cRIO to incorporate an internal PID control.

We replaced the classmate with one of our development laptops, a run of the mill Dell or Toshiba, to resolve our connector issue.

Ultimately these changes resulted in a flawless performance of the robot at Oklahoma, but does not necessarily provide any answers to our previous problems. It is evident, however, that we had at least two problems (loose ethernet connector and no CAN error handling) that should have been addressed before Kansas City and potentially would have quenched all of our problems. However, it was not worth potentially ruining another regional to test that theory out.

We may attempt to do more testing next year with the CAN control. Additionally we found that during testing in the build season, the laptop was never operated in practice mode (so we never truly simulated "field" conditions) and we never used the classmate in testing (potentially our biggest mistake).

Regretfully, after observing numerous Jaguar failures for other teams, and a few of our own, we may end up moving to the more bulletproof Victor controllers unless a hardware change is implemented in the Jaguars to improve their reliability.

Re: Robots not under driver control- does it happen- do you determine why?
 

After significant trouble at BAE which we primarily chalked up to using 2CAN and seeing many CAN timeout messages during every match: For Boston this week, we had moved to PWM.

We also pulled a chain and isolated a tan Jaguar that would work in forward but not in reverse.

We moved our radio to a higher place on our robot.

Control worked perfectly through all qualifying matches and three quarter-final matches. HOORAY!

In the first semi-final match that we played, about 30 second in we lost complete control- we rebooted the robot from the driver station and regained control for the last bit of the match.

The Field Tech immediately came over and started to help us try to figure out the problem. His initial hunch was a bad power wire to the cRIO. I thanked him and told him we had rebooted from the DriverStation. At that point, he asked if we changed our software, which we had because of a change requested for autonomous. He then suggested to run a simulation match while we waited for the other semifinal match. We did: and it ran with no problems.
We visually inspected and yanked on the cRIO wiring with no impact.

We played the second match and the Field Tech stood next to us for the entire match. We again lost communication for 20 seconds. This time we did not attempt to reboot from the Driver Station and got control back more quickly. The Field Tech at that point was fairly confident that it was a power to the cRIO problem.

And he was exactly right!

After the match, in the lighted area of the pits, after wiggling the power connections to the cRIO for about a full 5 seconds: the cRIO lights blinked and it reset.

SPEND TIME GETTING YOUR cRIO POWER CONNECTIONS RIGHT and TEST them periodically!

My advice: Use Checklists rigorously!

Re: Robots not under driver control- does it happen- do you determine why?
 

Twice during the elimination rounds in the VCU regional this weekend our robot would not respond for roughly two seconds. This had not happened at all during the qualification rounds or in our previous regional. Everything was fine otherwise, though, so I'm not sure what I should blame it on. Maybe some testing will help clear that up.

Re: Robots not under driver control- does it happen- do you determine why?
 

Dick, can you give more info on how the connection was done, and what was wrong with it. The reason I ask is, I hope to get an NI engineer to write a best practices article and specifically include this connector. I want to include images of common approaches and explain what is likely to fail.

Thanks,
Greg McKaskle

Re: Robots not under driver control- does it happen- do you determine why?
 

I'll post pictures next week after we get the robot uncrated.

My imperfect memory believes the stranded wire was not twisted tightly nor was it pushed in deep enough and thus sort of connected mostly as a right angle into the connector- making contact only with the top contact and the middle contact (by the sides of the blades but not the knife edges) and did not make it down to the bottom contact.

After I had wiggled two wires roughly for 5 seconds and got it to lose power, I was then able to pull one all the way out and its length into the connector was much shorter than I expected.

I'm always debating with myself if these types of connectors are better with a light solder to hold the twisted wire or the greater mechanical flex of a tightly twisted wire. I'm fairly certain a poorly twisted wire is a bad choice.

Re: Robots not under driver control- does it happen- do you determine why?
 

We get great results with no specific twist on a freshly stripped wire of the proper length. I firmly believe that solder is an inferior choice for both the WAGO connectors and the screw-clamp style on the cRIO.

I saw three cRIOs with power problems so far this year. One had almost no exposed conductor on the common wire and the connector was biting down on insulation. A tug test didn't reveal the problem, but it was obvious from a close visual inspection. The other two both had wires that easily came free from the connector when pulled. It looked like they were of thicker gauge than the connector could handle well, and only a few strands made it into the slot while the rest of the wire just sort of bunched up at the opening.

Re: Robots not under driver control- does it happen- do you determine why?
 

This sounds a lot like an issue in the code where it does not handle all the various States the robot goes thru in the transition from Autonomous mode to Teleop mode.

In Robot Main, the call to the "Get Robot Competition Mode" VI yields 2 key outputs: Robot Mode (Autonomous Enabled, Teleop Enabled, etc), and Call Context (Init, Execute, Stop). The state transitions from Autonomous to Teleop look like this (they are listed as Robot Mode / Call Context):

• Autonomous Enabled / Execute
• Autonomous Enabled / Stop
• Autonomous Disabled / Init
• Autonomous Disabled / Execute
• Autonomous Disabled / Stop
• Teleop Disabled / Init
• Teleop Disabled / Execute
• Teleop Disabled / Stop
• Teleop Enabled / Init
• Teleop Enabled / Execute

Note: I don't know how many cycles of Autonomous Disabled & Teleop Disabled the code will see. Under FMS control, it depends upon how long a gap if any there is between Autonomous & Teleop. Perhaps one or the other may not even happen.

Here's what may be causing the problem you're seeing. In Autonomous Enabled / Execute, your code sends a command to a Jaguar to move. When the Autonomous period of the game ends, your robot becomes disabled. That starts the chain of state transitions above. So the next state your code sees is Autonomous Enabled / Stop. In that state, your robot is already disabled, so the Jaguars are receiving no signal - hence their motors stop.

For some period of time, the robot sits in Teleop Disabled / Execute. When the Teleop period starts and your robot is enabled, the code 1st sees a Teleop Disabled / Stop, and then Teleop Enabled / Init. During both those states, the robot is enabled. The Jaguars will attempt do whatever command they last heard. Hence the start of Teleop has your robot moving the way it was last told to move in Autonomous.

One way to fix this would be to send a Stop command to all the motors when the robot is Disabled. This is already being handled in the default code by the Disabled VI - but only for the 2 default Drive motors. You need to go in and add a Stop for all your motors there.

Re: Robots not under driver control- does it happen- do you determine why?
 

We saw several instances of this at Smoky Mountain. Every case was eventually concluded to be a robot not exiting autonomous mode correctly, stopping their code at the end of autonomous, CAN errors, ethernet cable coming loose on the driver station, or intermittent power loss causing the cRIO or another key component to reboot.

Check your code by simulating a match on the driver station. If your autonomous functions properly in practice, it will execute on the field. Check all of your power connections on board and be sure that the main breaker is pushed in until it clicks. It is possible to close the breaker enough to allow power without it locking closed. Always check that the battery terminals and the primary lugs on the PD are tight. Sometimes the insulation covering them makes teams believe they are good when there is only intermittent contact. Lastly, I know that many of the driver stations develop issues with loose ethernet adaptors. If needed, use a small piece of duct tape or similar to temporarily hold the plug in just to be safe.