Robots not under driver control- does it happen- do you determine why?

[boomergeek]

Originally Posted by ChrisH

If you are at an event get with the FTAA before your match and ask him to pay attention to your robot. He has a display and logs of the communication status between the field, the robot, and the driver station. He can also spot a big drop in battery voltage, which indicates other problems. The logs can be checked but it is easier to just catch stuff as it happens most of the time.

I could not find this information in any FIRST user guide I'm aware of.
I would guess that the best/most experienced teams already know to do this. Unofficial information flow can sometimes unintentionally give an advantage to those in the know versus those trying to rely on official channels and not be a burden by asking for types of help.

A typical relatively new team sees the FTA as extremely busy with general responsibilities and focused on getting connections from all 6 DS to all 6 robots at the beginning of a match. The FTAs have to choose where to put their focus and if every team that had occasional control problems asked the FTAs to monitor their robot in every match, I think the FTAs would be overwhelmed: (this non-statistically accurate polling points in that direction).

For example, at GSR, when CAN timeouts seemed to percolate up as an issue: teams known to have CAN were informally polled by teams or by the FTAs: not all teams that had CANs were advised of the seemingly common issue in a timely manner- some played through Saturday with CAN while many teams (especially the better informed teams) had switched over to PWM by Friday
Obviously there is a balance of discretion about conclusions, but statistical observations by FTAs should probably be shared with all the teams at the same time, not just all those that ask or are in the know. It's also very clear to me that the FTAs are always trying to be as fair and generous as possible. Now the CAN timeout issue was probably a special case that does not happen very often. Working with the FTAs, the experienced teams form their own informal networks of FMS field issue information.
If several teams make a decision not to use a feature based on that collected information, it would be nice if the FTA broadcasts it in a timely manner.


Can someone post a picture of the real time log an FTA has available to them ?

Does FMS take requests for enhancements? How about logging all the detailed data per unit time and storing by team and match number?
Then each team could request only their data at any point during the competition. Also, with all this data logged, enterprising FTAs could look for issues and publish general results on problems teams are having.

Does the FMS logging have signatures for each of these problems that can be shared with everyone?

1) Radio placement
2) Radio proper cord and strain relief
3) Charged battery
4) shorting motors
5) software errors that cause UserProgram to run too slow (all the time)
6) software errors that cause UserProgram to run too slow (occasionally)

Can the same logging that FMS provides be made available in the driver station or the FMS light code?

Many teams work hard all season for only about 25 minutes of total actual playing time. (Some even less if their robot doesn't make the practice matches on Thursday).

I'm on a mission to reduce control problems for everyone especially the nubies.

A more rigorous (and scientifically accurate) poll at competitions to size the problem is probably the first step.

[willishf]

Having pneumatics in the on state and getting a momentary glich that turns it off causes a very dramatic movement that is visible. If a motor is spinning and you get a momentary toggling of output you won't see it.

Since more than one group seem to have this problem we can probably eliminate or zero in on common elements to find the source.

The problem appears to occur in both C++ and Java so not a language issue.

Camera and bandwidth issues

Wireless communication

Optical encoders generating interrupts that have a higher priority that need to be serviced and the watchdog timer doesn't get serviced. With two encoders and two inputs each 4 x 720/780 = 3000 interrupts a second. I am using 1X on the encoders which I think is serviced by the processor vs 4X which gets serviced by the FPGA and based on other forum discussion doesn't work with more than one encoder. I am under the assumption that the encoders are serviced as interrupts by software in 1X. The encoders give 720 points of resolution but when I rotate our wheel 10 times by hand I get very close to 7800. I have performed this test multiple times on two different wheels and aways get 7800+/-small error. If the resolution per turn was 720 then I would expect 7200+/-small error. I can work with the 780 but just indicates something isn't working as expected.

In looking through the JavaDoc more than one method to set watchdog and safety related attributes. Does anyone have any details regarding the need to set these parameters or does the default template work assuming you feed the watchdog?

[dbeckwith]

I noticed last year when we were using LabView that running the program from LabView and not as a start-up program significantly slowed down response and even caused some dropouts. I think this was just because the program was running on the ClassMate, so any time it slowed down (frequently!), the code slowed down. Also in LabView, the camera seemed to both be very slow and slow down the rest of the code, to the point of dropouts and even errors sometimes. This year we switched to Java and all those issues went away (mainly because the code is always a "start-up" program, it is run on the cRIO independent from the computer).

[Alan Anderson]

Originally Posted by willishf

Optical encoders generating interrupts...

Encoders do not interrupt the cRIO. They are always handled by the FPGA. 4x encoders use dedicated quadrature circuitry, while 1x and 2x use dedicated counter circuitry.

Originally Posted by willishf

The encoders give 720 points of resolution but when I rotate our wheel 10 times by hand I get very close to 7800. I have performed this test multiple times on two different wheels and aways get 7800+/-small error. If the resolution per turn was 720 then I would expect 7200+/-small error. I can work with the 780 but just indicates something isn't working as expected.

Where did you get encoders with 720 counts per revolution? The ones in the Kit of Parts this year are 360 cpr, and 250 cpr encoders are common. Is the wheel on the same shaft the encoder is on, or is there a chain and sprocket? Has the "distance per count" for the encoder been set in software, or is it left at its default value?

[willishf]

Thanks for clarifying the interrupt issue as currently focused on figuring out the non-responsive robot problem. I haven't had a chance to look through the cRIO docs to see what is under the hood.

The encoder is on the shaft of the gear box provided in the KOP. So a single wheel turn is probably giving a 2X increase on count. I initially went with 360 as one revolution of encoder and 720 for one revolution of the wheel. I have two encoders one for each wheel. I set the distance per pulse and in distance traveled simply didn't match distance measured. Put the robot up on blocks and turned wheel 10 times and get raw pulse count. Both wheels in the test indicate 780 pulses per wheel revolution.

We ended up building a second identical robot and ordered two additional optical encoders from andymark. Same exact results in that it appears to be 780 pulses per wheel revolution.

In theory the gearing ratio could be 2.166667 from the wheel to the rotation of the optical encoder shaft and the reason for the non integer multiplier. Always feel better when you have nice integer multipliers.

[pfreivald]

We're still trying to figure out why we're getting COM drops.

Our wiring is sound (and connections both strain gauged and either hot glued or taped).
Our power appears to be constant.
Our bridge was replaced with another bridge.
Our bridge is up high and visible everywhere but the bottom (it is upside-down) from 360 degrees with very little obstruction from the aluminum frame.
We are not getting errors as we run in autonomous or teleop.

...and yet, a few times at FLR, and a few times during practice, our bridge likes to reboot, killing our robot for about a minute.

Suggestions, anyone?

[Mark McLeod]

Originally Posted by boomergeek

Can someone post a picture of the real time log an FTA has available to them ?

This is a simple live display that the FTAs/FTAAs/CSAs use.
It's actually pointed at the field and can be video taped if you want, as long as no referee happens to be blocking it.

The logged data is much more intense and the FTA generally just checks it for obvious issues, such as low battery voltage and communications delays and drops. It is by team/match/etc. as you'd expect. Sorting through the multiple potential causes is more what the CSAs back in the pits with the teams do to relieve the strain on the FTA.
Generally, if an FTA sees a team having a problem on the field, he/she will ask a CSA to follow the team back to the pits to help troubleshoot.
Ultimately, it's vastly easier to learn to troubleshoot the problem from a volunteer who's already seen similar problems on 60 different robots.

The time to review the recorded match data is limited, however, because of the ongoing matches. It's much easier to ask the field crew to monitor it for you as your match is happening.
It doesn't have to be the FTA that watches it for you. Regionals have a help desk of some sort. I worked the NJ Regional, for instance, and we had about seven people available (counting FTAs), any one of whom could monitor the feedback for obvious issues.

You can also log your own data on either the cRIO or Driver Station.
Battery voltage, packet timing, etc. are easily collected.

[Mark McLeod]

Originally Posted by pfreivald

We're still trying to figure out why we're getting COM drops.

Our wiring is sound (and connections both strain gauged and either hot glued or taped).
Our power appears to be constant.
Our bridge was replaced with another bridge.
Our bridge is up high and visible everywhere but the bottom (it is upside-down) from 360 degrees with very little obstruction from the aluminum frame.
We are not getting errors as we run in autonomous or teleop.

...and yet, a few times at FLR, and a few times during practice, our bridge likes to reboot, killing our robot for about a minute.

Suggestions, anyone?

Just the obvious.

If your radio drops out for more than a minute, before it comes back, then it's not a position problem, it's a power supply problem.

• Watch the low point of battery voltage drops while the robot is under strain. Consider adding an error message to the Driver Station whenever the battery voltage drops below 6v or so.
• Verify the radio is on the special power protected PD output, of course. Found way too many on the camera 5v feed or regulat red/black wago connectors.
• Wiggle the barrel connector while under power in the pits to see if you can force a power outage. That's how we identify poorly fitting connectors (Radio Shack :-) ).
• Inadvertent contact with the button on the side of the radio.

[Alan Anderson]

Originally Posted by pfreivald

We're still trying to figure out why we're getting COM drops.

Our wiring is sound (and connections both strain gauged and either hot glued or taped).
Our power appears to be constant...
...and yet, a few times at FLR, and a few times during practice, our bridge likes to reboot, killing our robot for about a minute.

Suggestions, anyone?

What power connector are you using? There seems to be a hint that the DAP's power pin might be a tad smaller than an apparently correct connector is designed for. If you aren't using the connector that came with the DAP, it might be losing contact due to vibration of the board inside the case.

[boomergeek]

Originally Posted by pfreivald

We're still trying to figure out why we're getting COM drops.

Our wiring is sound (and connections both strain gauged and either hot glued or taped).
Our power appears to be constant.
Our bridge was replaced with another bridge.
Our bridge is up high and visible everywhere but the bottom (it is upside-down) from 360 degrees with very little obstruction from the aluminum frame.
We are not getting errors as we run in autonomous or teleop.

...and yet, a few times at FLR, and a few times during practice, our bridge likes to reboot, killing our robot for about a minute.

Suggestions, anyone?

Mike Betts posted something very interesting that might be related to your problems: Is it the right plug?

http://www.chiefdelphi.com/forums/sh...ad.php?t=93657

...I was a robot inspector at the Florida Regional last weekend. There were numerous complaints of momentary loss of communications but no single clearing house to vet all problems… I helped quite a few teams and came to the conclusion that no robot which used the power supply cable which came with the DAP-1522 radio had this issue. In fact, while working with one team, we succeeded in getting the DAP to be completely unpowered even though it appeared securely plugged in. We were able to restore power to the DAP by touching the power connector ever so slightly (virtually a feather’s touch). ...

[Hugh Meyer]

Originally Posted by Mark McLeod

You can also log your own data on either the cRIO or Driver Station.
Battery voltage, packet timing, etc. are easily collected.

Mark,

How do we get the packet timing? We are using C++.

-Hugh

[pfreivald]

Thanks for the suggestions. Here's what I've got so far:

1. Our voltage certainly is not dropping below 6V. We've monitored it closely just to make sure.
2. We are using a WAGO connector to plug the 5V inverter into the dedicated, protected PD 12V output. The right-angle power connector that came with the radio is connected to the radio and taped in with electrical tape wrapped around the entire radio to keep it snug.
3. Attempts to force a power outage by wiggling any of the connections have not worked.
4. Our radio is set up such that the switch will not be hit by anything, and we are 99.9999% certain that it was not hit by anything at any point where we lost communication.

We're going to re-wire everything anyway and test it on our practice bot before Championship. Hopefully whatever the issue is, that solves it.

Thanks for the help, folks!

[Alan Anderson]

Originally Posted by pfreivald

2. We are using a WAGO connector to plug the 5V inverter into the dedicated, protected PD 12V output. The right-angle power connector that came with the radio is connected to the radio and taped in with electrical tape wrapped around the entire radio to keep it snug.

Perhaps your 360 CPR 5v converter is flaky. Try banging on it and see if you can prompt a power outage that way.

[pfreivald]

Originally Posted by Alan Anderson

Perhaps your 360 CPR 5v converter is flaky. Try banging on it and see if you can prompt a power outage that way.

Good call. I'll try that!

[Mark McLeod]

Originally Posted by Hugh Meyer

How do we get the packet timing? We are using C++.

The easiest most efficent way is to just take the summary from the DLink statistics page. At least that's what I prefer to use since the DLink is made to collect that kind of data. It's nice because it's also the point that divides the transmission statistics into wireless (DLink<->field router<->DS) and Ethernet (DLink<->cRIO). We actually get additional results that the FMS logger doesn't have. In the past I've been able to identify a specific bad pin on the cable between the bridge and the cRIO this way.

Either captured by a query script (maybe while disabled after Teleop is over), or just examined via a web browser after a match. To examine it after a match be sure to leave the robot powered to preserve the match data until you can hookup your laptop.
Of course, if your DLink is rebooting then the data does get lost, but at least you'll have proof that it's rebooting.

I'd hesitate to log too much data too quickly on the cRIO, because of the potential for overload
If you're running a more powerful laptop as the driver station, logging the majority of the data back there might have less system impact, as well as being more easily accessable.