2007 Radio link reliability problem

[Dave K.]

Although there has been some discussion of this issue in this thread, I wanted to highlight the fact that for those teams experiencing radio communication problems, that you need to contact IFI... don't just sit and wait for a solution.

This past Saturday our team had enough of the robot completed to start some inital testing of the drive train and control system. We immediately began seeing dropouts in communication every 60-90 seconds.

Monday, we took a closer look at the issue and found that the problem is at its worst when motor electrical noise is at its worst. Although much easier to reproduce when the motors are under load, we were able to reproduce the problem with the robot up on blocks by making frequent directional and speed changes in the motors.

Testing showed that the problem appeared to become a bit worse as the primary battery discharged. Realizing that at some point, the sudden current demands placed on the primary battery may cause the control system and radio to fall back upon the NiCd battery, I substituted the NiCd with a bench supply set to ~7.2V and noted that that this substantially reduced the number of communication errors. So something as subtle enough as slightly lower power supply impedance on the backup battery input seemed to have an effect on the problem.

Throughout all of the testing, we swapped in previous years radios and had absolutely no communication problems. As soon as the 2007 radios were reconnected, the problems returned.

The problem we noted manifests itself in this manner:

- The Robot Controller's "Radio Modem" LED will flash yellow. As it does, the robot is not receptive to Operator interface input.
- Usually, but not always, the communication link itself may be lost for 15-20 seconds while the OI and RC resync. During that period the "Radio Modem" LED may alternately change color between yellow and green more than once.
- Towards the end of the syncronization process, the "No Data/Radio" LED on the OI will change to red briefly, and after it turns back to green, the link is re-established.
- On the RC, when the link re-establishes, the "RC Mode" LED on the RC will change from green to Yellow (disabled) and then back to green.
- The link is now re-established.


Monday night, after verifying that the problem was tied to the 2007 radio, I sent a note to Corey at IFI along with the details of my observations. When I asked if they wanted the radio's overnighted to them versus a ground shipment, he indicated that they have only received complaints from 4 other teams, and to date have only received 2 radio pairs back at their facility for analysis.

I shipped our radios back last night and Corey confirmed receipt today and that they are taking a look to see what they can do.

My intent in posting to a new thread is not to create any sort of panic, because it appears that this problem may not affect everyone. My intent is to highlight the fact that of the 4 teams that had notified IFI of a problem, apparently only 2 have returned their radios. Our pair would make it 3 out of 5.

If there are other teams experiencing problems that have not yet contacted IFI, please pick up the phone or send them an e-mail, and work with them directly so that this issue can be resolved as soon as possible.

IFI's contact information can be found here: http://www.ifirobotics.com/

Thanks,

[StretchyAlien]

While attempting to fix this problem, we may have accidentally stumbled over the answer. Our cell phones appear to be interfering with the signals between the radios. Try placing your cell phone near the radio and calling it, and see if this causes the interference. The "random" timing of the interruption may be the cell phones reconnecting to their respective service towers.

[RyanN]

Even more reason to hate this year's Radios. I've known of this problem since the first thread with the loss of communication. I hate this problem, and I hate how they are so huge. We are having trouble finding a safe place to put it with their fragile antenna. Grrr.... IFI...

[Gusman1188]

We dealt with this problem last year alot. In the end we started replacing battery, backup battery, cables between the modems and rerouting the modem cable away from motor power cables. It eventually started working after a while and it seemed like it just kind of fixed itself.

[roboticsguy1988]

Originally Posted by Gusman1188

We dealt with this problem last year alot. In the end we started replacing battery, backup battery, cables between the modems and rerouting the modem cable away from motor power cables. It eventually started working after a while and it seemed like it just kind of fixed itself.

Well as stated in the other thread, we originally thought ours was static problems. But after trying many many possible solutions, the radio data connection problem jsut sort of faded away after a while.

I will have to try connecting last years modems and see what happens. (Even though it wouldn't be legal to use this year, it would still help find the solution to the problem)

[Al Skierkiewicz]

Dave,
You are correct in your assumption that the main battery regularly falls below the 8 volt cutout for the RC ( we have seen robots that take this down below 4 volts on pulses). Generally a condition like reversing the motors will cause this as they fight to slow down momentum and then traverse a stall condition to change direction. However these are usually short duration pulses that the power supply in the RC can weather just fine. Since the modems are powered by the backup battery they should be relatively immune to power fluctuations on the main battery. The RC is a different story and I have seen teams that can improve life support for the RC by simply rearranging things in their electrical design. Frequently teams will take off power for the RC at the end of a long string of wiring that supplies power for high current devices. By feeding the small breaker panels with their own #6 input and return wiring (from the power distro blocks) and by using one of the first breaker positions to feed the RC, low voltage can be reduced. With four and six motor drives, it is very easy to loose 2 volts in the wiring and battery internal impedance on a fresh battery.

[billbo911]

We saw this problem occur last night with no load on the motors except the BB 56mm transmissions. We were running in the transmissions and testing the radios in one process. The battery voltage was monitored regularly and the lowest we saw was 12.1 vdc. Intermittently the motors would stop for just a fraction of a second with no variation in the input to the OI. We also saw the "Radio Modem" LED on the RC flash red at the same time the motors pulsed off and back on.
Bottom line: We will be contacting IFI today to report having the modem issue and wait for further directions from them on how to resolve the problem. In the mean time, we can use a tether or our 2006 modems to continue our build and testing.

[Al Skierkiewicz]

bill,
The voltage dips are relatively short so you would not see anything on the OI voltage monitor unless the dip occured at the exact instant that a voltage sample was taken or the event lasted longer than a few sample periods..

[waialua359]

We are one of the teams experiencing the problems. IFI hasnt got our radios back for analysis because its coming from Hawaii. We sent them out the same day we got our new ones.
UPS next day air came 3 days later and they gave us a UPS ground label to return. Hope it gets there soon.
Anyhow, this seems to be a big issue. If all of the suggestions made may be the problem, then I still feel the radios should be redesigned and/or they allow us to use last year's radios, if possible. I cant see how a previous radio would give any team an edge.
I cant be worried about cell phones, etc. during real matches at competition. We need reliable radios that wont lose signal at any time during any match at all.

[StretchyAlien]

Agreed. if it is the cell phones causing the problem, there is realistically no way to force everyone in a competition venue to turn theirs off. Our team had the initial problems as stated above, but then after a while, they seemed to go away unless we planted a cell phone on the robot and called it. Its almost like we are breaking in the radios the same way you break in the transmissions. Run them for a while and they are fine. Odd..

[Dave K.]

Originally Posted by Al Skierkiewicz

Dave,
You are correct in your assumption that the main battery regularly falls below the 8 volt cutout for the RC ( we have seen robots that take this down below 4 volts on pulses). Generally a condition like reversing the motors will cause this as they fight to slow down momentum and then traverse a stall condition to change direction. However these are usually short duration pulses that the power supply in the RC can weather just fine. Since the modems are powered by the backup battery they should be relatively immune to power fluctuations on the main battery. The RC is a different story and I have seen teams that can improve life support for the RC by simply rearranging things in their electrical design. Frequently teams will take off power for the RC at the end of a long string of wiring that supplies power for high current devices. By feeding the small breaker panels with their own #6 input and return wiring (from the power distro blocks) and by using one of the first breaker positions to feed the RC, low voltage can be reduced. With four and six motor drives, it is very easy to loose 2 volts in the wiring and battery internal impedance on a fresh battery.

Hi Al,

With the bench lab supply connected to the backup battery terminals, I did not notice any current being drawn while the RC was powered by the primary battery, but if it was less than 20mA it may not have been noticed on the 3A analog meter movement. Since the radios will work without a backup battery connected, my assumption is that the backup battery is only being tasked as the primary sagged below the backup batteries threshold.

As to our wiring, it couldn't get much shorter. We have about 12" total of 6AWG between the battery and Maxi-Block, with maybe 6" additional on the positive side to account for the main circuit breaker.

Each of the 2.5" CIM speed controllers negative leads are 10AWG return to the Maxi-Block and are also in the 6" range. The positive side connections between the speed controllers and the quad 40A blade type fuse/breaker block are probably no more than 6", and the input to the fuse block is 6AWG again about 4-6" long. The CIM's then directly attach to the speed controllers using the manufactuer supplied pigtails, which look to be about 14AWG.

The robot controller was the only thing tied to the smaller ATC fuse block, and the connections to that fuse block are also 6AWG connections back to the Maxi-Block with lengths on the order of 8-10".

Other than the Maxi-block compression connections, with one wire per clamp, and the 1/4" spade connections on the RC and smaller fuse block, all other connections are ring lug type compression fittings that are crimped/compressed and then soldered (in that order) to not only reduce impedance, but also to avoid vibration induced failures at a compression only junction.

All in all, the wiring could not be made much shorter than it already is and the only way to further reduce impedance would be to start eliminating some of the 'required' components, nor do I believe that increasing the wire size would substantially improve the electrical noise situation.

Recall that something as subtle as replacing the NiCd pack with a bench supply improved the situation, but did not eliminate the problem. Without measuring the supplied AA NiCd, I would presume that the ESR is already on the order of 0.1 Ohms.

Something that I did not try was adding some .01uF caps across the output terminals of the Victors to see if that improved the situation at all. Without opening up the motors (not permitted) to add them close to the brush assembly, we couldn't get them any closer to the motor without cutting and splicing the motor wiring.

[Al Skierkiewicz]

Dave,
It sounds like you have done everything I could have suggested. I am glad you are using #10 for the Chalupas as many teams try to get away with much smaller wire. We also solder our crimp connectors by using uninsulated crimps and heatshrink.
It is my understanding that the Radio modem and RC have a switched power input and watchdog circuit. When the primary voltage falls below 8 volts for some length of time, the two devices and all PWM power switches over to the backup battery. This keeps the RC from rebooting and keeps the modems in communication. (about a four second process after power on.) Unfortunately, four motors in stall coupled with the resistance of the main battery and the series resistance (as small as it is) will combine to drop voltage when current is being drawn. With four small Chalupas at stall, you could have as much as 400 amps of current flowing for short periods of time. That translates to as much as 1 volt drop when the battery is fully charged. If you have a scope compare voltage at the battery terminals and then move the probe and ground lead to the input of the RC or Victor. Run your motors and then reverse them and see the dip in the input voltage. I think you will be surprised at the difference. As to adding caps to the output of the Victor, this move is not legal, but was suggested as a fix last year. The Victor has some suppression built into the motor outputs. (Teams were reguired to add Varistors to the inputs on the old 883 controllers) Several years ago we had designed a custom circuit we called StangSense. It was a multichannel current monitor that would record the current in several loads and port it out the dashboard along with battery voltage and timer tics. This allowed us to analyze robot current and voltage while having a rough idea of what events in a match caused the data to change. We were not ready for what we saw. We made a portable version which we used to analyze other robots and the data was so accurate we could predict robot controller failures for teams. At that time it was a real issue since the RC was pre backup battery design. We regularly found dips in RC voltage below 4 volts on some robots including our own. That prompted a major change in wiring layout and wire size design as well as connector choice and soldering.
All things being equal, I think you have minimized the interaction. We are dealing with an unknown radio at this point and I don't think we fully understand what these new radios are capable of. I am sure we will have more data in the next week or so.

[waialua359]

Our new radios work perfectly now!
Thanks to IFI replacing the radios that came with our kit.