Comms issues on the FMS (PLEASE HELP US)

Hey,

We’ve had trouble connecting to the FMS at the Denver regional, which gave us a message in the log:
info communications timeout: # input voltage brownouts 0

the number seems to be how often it happened so far, and it always says 0 voltage brownouts, even if we had any brownouts. It also doesn’t fully drop communications, it only loses a few extra packets.

We usually got this message 10-20 times per match.

All controls seem to be running fine with the exception of our PID target changer, which is probably different because we send that command on the release of a button, rather than every iteration of the periodic. We’re using a network switch connected to a raspberry pi, which we don’t believe would cause this problem, but we’re open to advise. Furthermore, we only get this problem when connected to the FMS, in home mode and while tethered we don’t get these problems, which leads me to believe it’s not the code, but perhaps a problem with the cables(ethernet, router power, etc.) on the competition robot.

Any help would be greatly appreciated.
Thanks.

If you haven’t already, talk to a CSA at the event. They are always a big help. You can also ask to see the FMS logs by going to the question box on the field and talking to the FTA. They might also have some suggestions.

1 Like

Unfortunately Denver Regional was a Week 4 event, so it’s a little late to talk to someone at the event. But @somethingsBruin this is definitely solid advice for the future. The CSAs and the FTA(A)s at events are always happy to help you figure out what’s going on with your robot and try to prevent issues.

Have you looked at the logs from your driver station during those matches? Was your voltage dropping down excessively? If so, then I would check all of your power connections and make sure everything is tight and the wires aren’t bouncing around. Often times your wiring is getting much more of a workout during matches than it ever does at home, due to the intensity of the matches as well as defensive play.

2 Likes

As suggested go to the logs in the drivers station. You are probably not getting brown outs otherwise that info message would be greater than 1. You should focus in on the graphs for lost packets and trip time which cover the communication path. Issues that have caused this for us in the past include:
1 bad network cable connection at rio or radio
2 loose radio power cable
3 radio power coming off 500 mA vrm circuit instead of 2 A vrm circuit
4 plugging anything else into sam vrm circuit
5 bad POE cable
6 flooding the comms with log error messages

You can certainly do that, but what we can show from the FMS point of view is the 1/2 second snapshots that your DS sends us throughout the match in a nice table form. I can look at that and see where to pinpoint on the DS log file to help sort out the chaff from the wheat in narrowing down the issue. All logging on the field comes from the DS.

@somethingsBruin, would you be able to send me a log file where the problem happens and I can take a look for you to maybe narrow down where to search on the bot?

We did talk to the CSA and some FTAs at the event, they tried to help but were as stumped by it as we are. Thanks for the reply though!

We have looked at the logs as extensively as I can understand them, unfortunately the robot is bagged and I can’t check the connections, but as soon as we unbag it I will check all of this. Thanks for the help. We did have quite a few brownouts during most of the matches, but it was due to crappy batteries. Could that have caused major comm issues without a full disconnect.

The logs do tell us that we’re getting brownouts, it’s just that the messages in the logs’ event list say that we’re getting Info Communication Timeouts: num, Voltage brownouts 0. Despite having multiple brownouts per match according to the same log. As to the rest of your advice, I will male sure to check those as soon as the robot is unbagged, I’m just a programmer, so I don’t know much about wires, so I won’t have info on this til it’s unbagged. Thank you so much for the help.

I can send you the log files, but I’m currently on my home computer. I can get them to you tomorrow by around 1:00 pm Central mountain time.
Thanks for the help.

OK, so if the logs are showing brown outs then you need to look at the DS graphs for each of the PDP slots and the PDP overall. If you see extreme spikes in one robot function i.e. drive slots concentrate there and you may need to limit draw using code and speed controller configuration. If there are no spikes it may be loose wiring on the battery side of the pdp/breaker of loose fuses for the vrm/roborio

we figured out the brownouts, we were using bad batteries but I’ll check those logs. How do i limit power draw programmatically? Thanks again for the help!

Sarcastic answer: Don’t run your motors.
Real answer: You cannot. Robots don’t have current limiters on them.

Non-sarcastic short answer: You monitor the system voltage at the PDP (aka battery voltage) and reduce/regulate the commanded output of your motor controllers to prevent the system voltage from dropping below some threshold.

The exact program details will depend on your robot design i.e. what motors you have on what functions and what functions can be sacrificed to keep power going to others.

The people with programming backgrounds here can help you with the details.

It will also depend on the motor controllers you are using, and if the problem is a motor max load or spiking during initial acceleration. Different strategies for each. E.g. if its an instant acceleration spike you can smooth that out using ramp rate. Versus a too big load at max speed where you could scale down the motor output so it never hits 100%.

If you’re using a “smart” motor controller like a Talon SRX or SPARK MAX, you can set limits on the continuous and instantaneous currents. So if you try to command the motor to pull more than the limited current, it will altomatically scale back the output until you reach the defined limit. And by limiting each motor individually, you effectively limit the current draw of the whole robot.

If you’re using “dumb” motor controllers (anything controlled by PWM) then you would need to do the current limiting manually. In that case, it’s probably easier to modify the system mechanically or lower the max voltage to lower the current rather than trying to roll your own current limiting software.

Common causes of lost packets:

  • debug printfs
  • calling an object set method that grabs a mutex to change the state
  • calls to objects to fetch data that may block or take a long time, like a gyro

If you have high cpu as well that points to debug printfs.

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.