Chief Delphi

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)
-   General Forum (http://www.chiefdelphi.com/forums/forumdisplay.php?f=16)
-   -   Possible FMS problems. (http://www.chiefdelphi.com/forums/showthread.php?t=127617)

Canon reeves 08-03-2014 20:34

Possible FMS problems.
 
Our team competed at the Arkansas Regional and we ran practice matches all Thursday, no problems at all. There was no score keeping during these matches and we did good. During our first qualification match we ran great until the last 15 seconds we lost power to the C-Rio, and all Comm. and Code. During the next match our battery fell out. And in every other qualification match we would run, then our C-Rio would reboot. Sometimes it would happen when we took hard hits, others we would just drive forward and then reboot. So first we replaced the C-Rio, same problem, all of the wiring was fine, had multiple inspectors confirm that, Then we replaced the PD board, took it to the practice field and it worked fine, we ran at full speed into the low goal multiple times, it was just fine. Then we went onto the field on saturday and same problem happened. We replaced the main breaker, same problem. So then we got picked for elims. and during the time in-between our match, we ran on the practice field with our alliance members at least 20 minutes, just fine. We went out on the field, drove a little bit, and re booted. same thing happened next match. Any ideas what could be the problem?

M. Lillis 08-03-2014 21:01

Re: Possible FMS problems.
 
Although I don't have any solution or hypothesis for your problem, I did notice something in one of our matches last weekend. The FMS messed up the hot goals (As seen here).

None were hot for the first ~5 seconds, then the left was hot for the last ~5 seconds. But right as the bell rings for teleop to start, the right turns yellow, indicating that it is hot. If you look to the left of the frame, you can tell (watch a few times) that the same thing happened for the blue alliance.

Tom Line 08-03-2014 21:03

Re: Possible FMS problems.
 
The FMS system this year could definitely use improvement, however it would not cause your robot to reboot. You have a wiring or power distribution problem that you still haven't discovered. They can be tricky!

Canon reeves 08-03-2014 21:06

Re: Possible FMS problems.
 
I never saw much of that during our regional, and the inspectors were absolutely great working tirelessly to diagnose the problem, it just seems weird because we replaced the all of the electronics and had the same problem and it was even worse in eliminations even though we didn't take a lot of hits? and we had just been running great? I'm just unsure, not upset, I just want to know what we need to fix.

Canon reeves 08-03-2014 21:10

Re: Possible FMS problems.
 
Quote:

Originally Posted by Tom Line (Post 1355534)
The FMS system this year could definitely use improvement, however it would not cause your robot to reboot. You have a wiring or power distribution problem that you still haven't discovered. They can be tricky!

While the C-Rio is rebooting, it isn't the only thing, in several matches the d-link did as well, and it seemed like we would randomly just shut down and not work, and under diagnostics the robot light would be green, not the bridge. I'm just wondering if there could be something in our code having a bad reaction with FMS? I would completely agree it is a power or wiring issue if we hadn't been running perfectly fine any time other than when connected to the field.

Canon reeves 08-03-2014 21:15

Re: Possible FMS problems.
 
https://www.facebook.com/photo.php?v...type=2&theater
here is a link to one of the matches we played in. We are the robot with the huge orange net, at around 1:55 in the video, we lost all communication.

MikeE 08-03-2014 21:24

Re: Possible FMS problems.
 
Quote:

Originally Posted by Canon reeves (Post 1355536)
While the C-Rio is rebooting, it isn't the only thing, in several matches the d-link did as well, and it seemed like we would randomly just shut down and not work, and under diagnostics the robot light would be green, not the bridge. I'm just wondering if there could be something in our code having a bad reaction with FMS? I would completely agree it is a power or wiring issue if we hadn't been running perfectly fine any time other than when connected to the field.

Did you work with the CSA (Control System Advisor aka "orange hat") at your event? They are experts in robot troubleshooting.

From your description it does sound like a power/wiring problem, possibly a connection to the radio or a short to the frame.

Take a look at the Driver station log viewer for the matches where the problem occurred. Details of how to do this can be found at the WPI screensteps pages

It might also be a good idea to cross post this in the Technical Forums.

Richard Wallace 08-03-2014 21:28

Re: Possible FMS problems.
 
^+1 to what Mike said above.

We had a similar issue in one match. Afterwards we found an intermittent short to our frame that had gone undetected during inspection. We fixed that, and never saw the problem again.

Canon reeves 08-03-2014 21:30

Re: Possible FMS problems.
 
Quote:

Originally Posted by MikeE (Post 1355549)
Did you work with the CSA (Control System Advisor aka "orange hat") at your event? They are experts in robot troubleshooting.

From your description it does sound like a power/wiring problem, possibly a connection to the radio or a short to the frame.

Take a look at the Driver station log viewer for the matches where the problem occurred. Details of how to do this can be found at the WPI screensteps pages

It might also be a good idea to cross post this in the Technical Forums.

Yes we did work with him, he was unsure after we replaced everything, we looked at the logs, definite signs of the C-Rio rebooting, and he also saw the D-Link reboot. The thing that I am confused about is how we can run just fine multiple times when we are off the field, but never work when we are on? I do not know how to cross post? Are you saying to post the same thing in a new thread under technical Forums? Thanks for the feedback guys! I don't mean to try to argue or anything, I just want to figure this out.

cgmv123 08-03-2014 21:33

Re: Possible FMS problems.
 
Do you want to post your logs from the matches where you disconnected?

tStano 08-03-2014 21:38

Re: Possible FMS problems.
 
I suspect an intermittent short somewhere in your system. We had a similar problem today with our practice robot, and that was the issue.

We also had a similar problem last year and could not figure it out. We found several things we thought were causing it, but it just kept happening. I'd be very interested to hear your solution if you find it.

MikeE 08-03-2014 21:38

Re: Possible FMS problems.
 
Quote:

Originally Posted by Canon reeves (Post 1355555)
Yes we did work with him, he was unsure after we replaced everything, we looked at the logs, definite signs of the C-Rio rebooting, and he also saw the D-Link reboot. The thing that I am confused about is how we can run just fine multiple times when we are off the field, but never work when we are on? I do not know how to cross post? Are you saying to post the same thing in a new thread under technical Forums? Thanks for the feedback guys! I don't mean to try to argue or anything, I just want to figure this out.

In my experience some wiring issues only become a problem in the hardest of hits. These tend to happen in the heat of competition and rarely on the practice field.

You could post your original question in a new thread in the Tech forums and link to this one. It's very likely the underlying issue is an intermittent & tricky problem with the robot so I'd suggest a different title to the thread.

Canon reeves 08-03-2014 21:39

Re: Possible FMS problems.
 
Quote:

Originally Posted by cgmv123 (Post 1355557)
Do you want to post your logs from the matches where you disconnected?

I don't have access to them now, but I will definitely post them as soon as I do. In no way am I trying to blame the field, we built the robot, the results are more than likely our fault, I'm just very confused and concerned for for other teams if they run into a problem like this. At first it seemed a power issue, then we made sure everything was good, then we just kept fixing things but it just kept happening, the CSA checked the logs every time showing the same thing. I will try to post those sometime this week.

Canon reeves 08-03-2014 21:49

Re: Possible FMS problems.
 
I made a new thread, and again, thanks for the feedback. http://www.chiefdelphi.com/forums/sh...hreadid=127623

cglrcng 09-03-2014 00:34

Re: Possible FMS problems.
 
When you next get to unbag that comp bot....Look at the Radio Power Converter (there are 4 wires....1 yellow, 2 black center ground wires, and 1 red).

Are the 2 center grounds possibly crossed over? The black nearest the red wire should be paired w/ the red. The black nearest the yellow wire should be paired w/ the yellow.....And the yellow should be connected to the radio power + side (black w/ white stripe wire to the radio).

We had no problems whatsoever w/ that exact type of rebooting issue (you seem to be alluding to), w/ our competition bot, then we put that bot into the bag happy as can be after much practice on the 18th of Feb....That 1 was wired correctly.

Started working w/ our practice bot later, since we had been waiting for a ball P.U. gearbox to arrive, and lo and behold after that was in and running right, we had the same exact "once in a while connection loss issue" that you speak of when testing the practice bot"....Shoot, dry shoot, load a ball (no problems, except a very intermittent loss of connection that got worse, not better). We zip tied the radio power connector, changed radios, checked for a frame grounding issue, we thought we did everything....Then, we noticed a real error in wiring (one of the students wired the radio power converter incorrectly, as the 2 center grounds were crossed over).

When we removed the power converter (holding it w/ wires away from you, wires top, from the left), the order should have been yellow /black-black /red...(Make sure they are paired just like that).

Wired correctly now (;-0), we have never lost connection again no matter what we hit, or do, after many rough hours of practice. (It was simply the 2 blacks crossed over...The black nearest the red was w/ the yellow...and the black nearest the yellow was w/ the red. Those 2 grounds are not created equal by any means on that power converter). Wasn't easy to notice as that converter was in a spot way down under and very hard to see unless you were looking very closely.

The power connector to the d-Link radios are also very famous for intermittent loose connections and subject to wiggling in the socket easily...tape it, zip tie it, etc. (but of course, you cannot solder it...You need to get that radio configured at the events at the kiosk, as you do already know).

Please post back if you find that is the problem please (It may not be, but ours was...and both act the same). And GOOD LUCK in tracking down an electrical gremlin, as they are not always easy.

Edited: BTW, that practice bot electrical board was the 1st 1 wired...(It was the practice board of course).

techhelpbb 09-03-2014 01:28

Re: Possible FMS problems.
 
Let's start from the bottom with this:

Please locate a multimeter with a Min/Max function, put it on your robot setup to measure minimum voltage DC and attach it to your battery lugs.

Then drive your robot around for 3 minutes (longer than a match) and see what the lowest voltage recorded is. Please post that here.

Then drive your robot around gently bumping stuff and see what the lowest recorded voltage is (similar to a situation with a field robot collision) and post that lowest recorded voltage here.

Generally a multimeter records voltages in Min/Max about 1,000 times a second. That's not bad considering the most (not all) critical electronic loads on the robot have additional capacitance which will ride through short drops.

Now once you achieve this, let's keep in mind that at high current you might have wiring somewhere that's too high in resistance (poor connections or an intermittent short). To test this put that same multimeter on the power inputs to these critical electronic devices (cRIO, radio, radio DC/DC converter, digital side car) again set to minimum voltage DC and repeat the tests above. Post that data.

With this in hand you are not depending on any aspect of your FIRST robot to measure the power supply critical to your robot. In fairness usually the driver's station charts will show voltage drops - but just in case there's something not right there this process I've outlined will show such problems as well.

Also I may have missed something but which programming language did your team use for your robot? C++, Java or LabView? Does your robot use vision?

Keep in mind that the field system does have some very real timing differences from other private wireless and wired robot situations. It's possible to cause yourself some headaches. This year I've seen problems with putting the Java camera get instance in the robot init section (this is a perfect example it only was an issue on the real field). I've seen problems with LabView and one team's attempt to use the standard compressor and pressure switch cause their robot to reboot consistently because LabView kept saying that relay 1 was an invalid index (clearly that's not an invalid index and strangely if you edit the block and set that directly it's fine for a bit and then the problem returns). Also with C++ you can easily shoot yourself in the foot (not so much a problem with C++ but just the nature having that much power it can blow up).

So I advise you even if you provide the voltages I request above to take your code and features apart starting at the most simple and adding on till you reach the full compliment. In this manner if one thing is causing headaches you can isolate which one(s) require more evaluation.

Foster 09-03-2014 09:44

Re: Possible FMS problems.
 
As always techhelpbb has a good set of steps to follow.

He asked
Quote:

Please locate a multimeter with a Min/Max function, put it on your robot setup to measure minimum voltage DC and attach it to your battery lugs.
Putting on my psychic problem hat (*) I'm wondering if there is a correlation between your battery falling off the robot issue and the power cable and attachment to the Power Distribution Board. I'm thinking that the match was not the first time the battery has come off the robot.

So I'd like you to Please locate a multimeter with a Min/Max function, put it on your robot setup to measure minimum voltage DC and attach it to your Power Distribution lugs. That way you will be able to see if there is a problem with your main power feed.

Report back with the stuff that's been asked in the prior threads, that should exactly pinpoint the problem.

Good luck!

(*)Yes, I said psychic hat. I know the difference since I have one of each. :rolleyes:

Bald & Bearded 09-03-2014 10:06

Re: Possible FMS problems.
 
To echo above. Check your software. When on the field several tasks on the CRIO interact with the FMS that pretty much sit idle when on a practice field.
If you have some sort of memory/buffer overrun or bad pointer in C++ that can cause that type of behavior.

RufflesRidge 09-03-2014 10:55

Re: Possible FMS problems.
 
Quote:

Originally Posted by Bald & Bearded (Post 1355779)
To echo above. Check your software. When on the field several tasks on the CRIO interact with the FMS that pretty much sit idle when on a practice field.
If you have some sort of memory/buffer overrun or bad pointer in C++ that can cause that type of behavior.

Software will not cause your radio to reboot. This definitely sounds like a power problem.

Secretspy97 09-03-2014 11:17

Re: Possible FMS problems.
 
We had a similar problem this year. During driver practice we would be able to run the robot for about 3-4 minutes. After that the cRIO would randomly reboot or we would pop the 120 amp main fuse. We checked if the frame was grounded or if any metal shavings were found in the PD board and found nothing. This problem continued multiple times afterwards but we still haven't diagnosed the problem because the next day was bag and tag day.

However after searching chief delphi for answers we found this: http://www.chiefdelphi.com/forums/sh...ad.php?t=91733

We use a Banebots 775 to tilt our shooter. It is directly mounted to our aluminum frame. Since the motor only shorts out as different spots. This can explains why you don't see any electrical problems when we test for conductivity to the frame. We currently think that this may be our problem.

If you use a Banebot I would disconnect power to it and see if you experience the same problem.

Jarren Harkema 09-03-2014 13:07

Re: Possible FMS problems.
 
Quote:

Originally Posted by Bald & Bearded (Post 1355779)
To echo above. Check your software. When on the field several tasks on the CRIO interact with the FMS that pretty much sit idle when on a practice field.
If you have some sort of memory/buffer overrun or bad pointer in C++ that can cause that type of behavior.

Quote:

Originally Posted by RufflesRidge (Post 1355790)
Software will not cause your radio to reboot. This definitely sounds like a power problem.

We were having similar problems with our radio/cRIO rebooting. We aren't exactly sure what happened, but usually right after auto or half way through teleop, we would loose comms for 15 or so seconds. The radio power light would go red, and then turn back blue. Upon looking at the driver station log, just before loosing comms, there were a bunch of errors relating to the camera. Simply unplugging the camera ethernet camera resolved the issue. Somehow we must have a memory leak in our vision code causing the cRIO to reboot. but its weird that the radio would reboot as well, as it would not have lost power.

Canon reeves 09-03-2014 14:09

Re: Possible FMS problems.
 
Thanks for the great advice guys, I think during one or two of the matches we did have a loose battery lug, and we also had a loose anderson, we found this out by shaking them while it was on and causing it to cut out. I will try this week to give yall the information. We use labview, I'm not sure what the code exactly looks like because I don't program, but I will get that info as well. The big question that has me stumped is how most of these problems that yall talked about wouldn't occur off the field, even when we ran into the wall vigorously a lot? We only verified a D-link reboot twice, and that was on matches that we had definite battery problems. By no means am I an expert on any of this so please explain why it makes sense. Thanks for the input!

Alan Anderson 10-03-2014 01:16

Re: Possible FMS problems.
 
Quote:

Originally Posted by Canon reeves (Post 1355855)
The big question that has me stumped is how most of these problems that yall talked about wouldn't occur off the field, even when we ran into the wall vigorously a lot?

One of your mentors asked me that several times, and my answer is still the same: running into the wall is a jolt in one direction. Being bumped by another robot from the side, or from behind, is another direction. If there's a loose connection that causes problems related to being hit, it won't necessarily show up with a hard impact to only the front of the robot.

The one time I was close enough to see the speed controllers' LEDs during one of the on-field episodes, the robot was definitely completely without power immediately after being bumped from behind. Some ten seconds later, a second gentle bump on the corner coincided with the power coming back on. Nothing in software or the field communication can cause that kind of failure.

Quote:

We only verified a D-link reboot twice, and that was on matches that we had definite battery problems. By no means am I an expert on any of this so please explain why it makes sense. Thanks for the input!
The D-Link rebooting was only verified visually twice (it was mounted on the robot such that the lights are only visible from a narrow angle of view). The real-time field data monitor showed a loss of radio communication every time I was able to watch, and the FTA says it happened every time your robot had a problem on the field, though the logs I was able to review were only clear about the cRIO reboot and were inconclusive about the D-Link (the Driver Station apparently doesn't ping the bridge during enabled mode, and the FMS stops recording when the cRIO is not responding).

The one "smoking gun" we identified was the intermittent SB50 connection. Right after that penultimate match, it was definitely easy to cause the robot to lose power by pushing on the red wire in the right direction. I understood that the connector was going to be replaced. Afterwards, when the robot had yet another verified radio reboot with associated logs suggesting a complete power dropout, I noted the same loose feel to the red wire's terminal. I didn't think to ask whether it was the entire connector and its wiring that had been replaced, or if it was just the housing, or if it was just the terminals and wires. It is possible that the red SB50 shell was reused, and that the underlying fault is that its "spring" isn't holding the terminal hard enough to the matching battery connector. Or it is possible that the terminal itself wasn't changed and has a bad surface that doesn't provide a secure contact.

Being unable to help 4490 correct the repeatedly-appearing problem is definitely something I feel bad about.

juchong 10-03-2014 03:23

Re: Possible FMS problems.
 
Try replacing the buck converter that's connected to your radio and make sure that the converter case is isolated from the rest of your robot. We saw quite a bit of that on the field at Hub City, as well. We verified that all the connections were done correctly, that the battery was properly secured, and replaced the radio (just in case).

BBray_T1296 10-03-2014 03:34

Re: Possible FMS problems.
 
Quote:

Originally Posted by Canon reeves (Post 1355509)
During our first qualification match we ran great until the last 15 seconds we lost power to the C-Rio, and all Comm. and Code. During the next match our battery fell out. And in every other qualification match we would run, then our C-Rio would reboot.

The FMS cannot cause your battery to fall out.
There is a good chance your wiring was pulled unnoticeably loose and caused a slight loss of power during some particular hits.

Possible causes: loose lugs somewhere between battery and PD board, loose/improper crimps/WAGO connections

Just because it looks correct, doesn't mean it is.

Greg McKaskle 10-03-2014 08:27

Re: Possible FMS problems.
 
The FMS doesn't directly communicates with the robot.

The FMS will tell your DS computer when to be disabled, auto, and tele modes, but it is your DS computer that sends all packets to the robot via the wifi radio on the field. The Practice match sends the same info. The code on the cRIO runs the same whether on field or in pits.

If your battery is able to fall out once, it is likely that it was loose at other times. When not held firmly in place, the battery is a 12 lb. sledge hammer banging into your other components. It easily knocks wires out of Wago terminals, knocks ribbon cables off, knocks wires and terminals against the frame, etc. As Alan said, its direction of travel will also depend quite a lot on the type of hit.

Greg McKaskle

Canon reeves 10-03-2014 10:09

Re: Possible FMS problems.
 
That makes a lot more sense, so what we really need to check next is our mainbreaker crimps, and connections, and our converter for the d-link, it is also very possible for the battery to mess up connections because we had our battery board fall down on at an angle and there was no visual damage. We will also try a good shaking of the power wires to see if we can get the problem to re occur so we can pinpoint it. Thanks for all the feedback guys, and a special help to Alan for all of his hard work and helping us find the problems!

Richard Wallace 10-03-2014 10:21

Re: Possible FMS problems.
 
Did each of the SB50 terminals "snap click" into its proper place within the housing; i.e., toes over the end of the diving board? Sometimes if the terminal is bent slightly, the "snap click" does not happen and the terminal can back out a little, just enough to make intermittent contact. A good whack could open such a contact, and another good whack later could close it again. This can be hard to find until the terminal backs out far enough to cause the connection to open statically.

Al Skierkiewicz 10-03-2014 10:26

Re: Possible FMS problems.
 
Guys,
There are two power supplies on the PD, one for the cRio and the other for the radio. while similar in design these are separate supplies. They will stop operating when the battery voltage input to the PD falls below 4.5 volts. Since they are separate supplies they may stop at different times. Loose battery connections, poor crimps, loose hardware on the battery and a bad SB50 connector can all contribute to losses that will shut down these supplies. However, even these supplies cannot stay up with no power to the robot. So, as inspector and electrical guy, I work this list...
1. Are the terminals on the battery tight and non-moving? Typical of carrying the battery by the wiring. Cannot be repaired, battery is likely candidate for recycling.
2. Do the wire terminals rotate on the battery terminals? Typical for these battery terminal types. The simple addition of a star washer between the terminals will prevent this.
3. Is the hardware tight on the PD? Tighten using metric nut driver.
4. Is the hardware tight on the main circuit breaker? Tighten using english nut driver.
5. Is the wire secure in the terminals used? Tighten and strip to correct length if needed. SLA connector in particular have an optimum strip length that is needed for secure electrical connection.
6. Is everything correctly insulated? Electrical tape, not duct tape is your electrical friend.

Canon reeves 18-03-2014 21:59

Re: Possible FMS problems.
 
So we have been running tests with a multimeter and it doesn't seem to be a PD board problem, and I replaced all of the power connections to the CRIO and when I got ready to drive I connected, enabled, then drove forward slightly and rebooted. The CRIO didn't reboot until enabled and drove? We also had the whole power go out, and I am not certain what caused that, it was most likely the battery because the Anderson was at a weird angle on it. Aside from that, could the problem be in the Ethernet cord from the D-link to the CRIO? Because we did tether when we got the robot back and it drove fine, so could it possibly be a bad Ethernet cord causing it? We went ahead and replaced it but could it have been the problem?

Alan Anderson 18-03-2014 22:29

Re: Possible FMS problems.
 
Quote:

Originally Posted by Canon reeves (Post 1361076)
...I connected, enabled, then drove forward slightly and rebooted. The CRIO didn't reboot until enabled and drove?

That could be caused by a bunch of things. Two seem likely enough to be worth investigating. Maybe your battery was essentially dead and couldn't provide enough current to run your drive motors. Maybe you have a couple of frame shorts that bring the cRIO power return to a high enough voltage that 24v on the power supply isn't sufficient to keep it running.

A bad Ethernet cable won't cause a cRIO reboot.

Fifthparallel 18-03-2014 22:32

Re: Possible FMS problems.
 
One thing I've noticed is that this problem seems very particular to the 4 slot cRio and will occur with very secure power connections. I've seen this happen to about 11 robots over the past two years, where the robot will be hit rather hard and will reboot for no apparent reason.

I even troubleshot this with a rookie robot where we discovered that if we dropped the robot (above 1" drop height) while the cRio was relatively not constrained to the robot, the cRio would reboot. Driving the robot at fast speeds and stopping quickly or tugging the wiring in different ways (which would have exacerbated any loose wiring) did not cause the cRio to reboot. However, driving at high speeds into any solid surfaces or getting the robot hit (even with a swift kick to the bumpers...) would cause the cRio to reset based on the force used.

Does the 4 slot cRio have a force-sensitive sensor inside of it that tells it to reset if a threshold is met? This seems much too silly to be the cause, but it seemed too consistent to not be the problem. It was even a problem for Team 1410 at the 2014 Utah Regional, where our cRio reset after hard connects with other robots.

Alan Anderson 18-03-2014 23:06

Re: Possible FMS problems.
 
Quote:

Originally Posted by Fifthparallel (Post 1361087)
...a force-sensitive sensor...

A main circuit breaker with a certain kind of fault can do a good job of detecting impacts in specific directions and briefly shutting off before returning to normal. A piece of metallic debris near certain components, or a stray bit of wire strand, can cause a loss of power when the robot is jerked in a specific way.

There really ought to be a better way of finding such intermittent situations than just banging and tugging on things at random, but when pressed for time anything more systematic might not be worth trying.

TheRamAlakazaam 19-03-2014 00:27

Re: Possible FMS problems.
 
Do you use a smart dashboard..
Our robot would reboot after autonomous because of it.
Turns out only Java Users can use it. So my team decided to set it to false along with other teams and we are all driving "blind" basically.

RobotKnight2014 19-03-2014 00:32

Re: Possible FMS problems.
 
At the Utah regional right? I was wondering if you ever fixed that issue, glad to hear you got it figured out.

Quote:

Originally Posted by Richard Wallace (Post 1355554)
^+1 to what Mike said above.

We had a similar issue in one match. Afterwards we found an intermittent short to our frame that had gone undetected during inspection. We fixed that, and never saw the problem again.


Greg McKaskle 19-03-2014 07:02

Re: Possible FMS problems.
 
The cRIO does not have a shock sensor. The data sheet showing the shock and vibe testing is here .. http://sine.ni.com/ds/app/doc/p/id/ds-354/lang/en.

If you are in a relatively static-free environment, you can clean debris from the cRIO in about five minutes. I almost always find lots of glitter and wire bits. I have seen cases where the reboots ceased for the remainder of the matches. I have also seen cases where nothing improved. Procedure link is below.

http://digital.ni.com/public.nsf/all...257673007935A1

Greg McKaskle

Richard Wallace 19-03-2014 07:02

Re: Possible FMS problems.
 
Quote:

Originally Posted by RobotKnight2014 (Post 1361144)
At the Utah regional right? I was wondering if you ever fixed that issue, glad to hear you got it figured out.

Our team is in Michigan -- the issue I mentioned occurred at the Southfield District in Week 1. The intermittent chassis short was caused by some wiring that had previously provided power to a custom circuit (LED string), which had been removed after it was damaged during a practice match.

What was the issue seen at Utah?

Al Skierkiewicz 19-03-2014 08:10

Re: Possible FMS problems.
 
Canon,
The issue you describe is usually attributed to one or more of the following problems in no particular order.
1. The cRio is bumping into robot frame or is attached to robot frame electrically.
2. Wire whisker is touching the opposite polarity on the power supply wiring at either the cRio end or the PD end.
3. Power wiring is not inserted into the connector properly, a tug on the wiring will show this problem.
4. Rare manufacturing defects have been observed on the main breaker. while the robot is thruned on and booted, trying lightly tapping the red reset button on the breaker. If the lights on the robot blink, replace the main breaker.
5. Improperly terminated battery wiring. Check everything from the battery to the PD. If it moves, it is intermittent.
6. An intermittent short exists in the branch wiring or you have a defective motor controller or motor that is showing a short in one direction. Try removing all breakers and reinserting one at a time to locate the offending branch.
7. If you are using multi motor transmissions, check that the motors are wired properly so that they are not running in opposite directions.
8. You have an intermittent power wiring to the DLink or you have not wired the power convertor to the dedicated +12 volt output on the PD.

When you have a shock intermittent, use a large screwdriver to tap the handle (or other insulated tool) on various parts of the robot until you find the sensitive area of the robot. You may find swarf in the cRio, PD or DSC or you may find improper wiring on the outputs of one of the cRio modules. If the condition only occurs when you are driving, the problem is in the drive train or the cRio to robot frame short.

Canon reeves 19-03-2014 10:01

Re: Possible FMS problems.
 
There has been a lot of mention of a frame short, could we test for this by testing the frame with a multimeter? we did that and it wasn't conducting anything, we checked the battery and all the connections to the pd board with it and got a consistent 12.5 volts. After the testing when the robot drove forward and shut off, we tried turning it back on, it took a few times but after it did turn back on we had no connection whatsoever, we tried rebooting, reset the radio, re-deployed code, exited the driver station then came back in. Under the diagnostic page on the driver station there was no light for the bridge, but there was a red light beside "robot". I've been trying to get it to shut off by shaking and dropping, and it still isn't rebooting like it is when I drive it. Is there anything else in the driver station we can use to diagnose the problem? Thank you guys so much, the team really appreciate's your help and GP!

noceradave 19-03-2014 10:06

Re: Possible FMS problems.
 
Check your event log from the match and see what happened:

http://wpilib.screenstepslive.com/s/...og-file-viewer

We had a similar problem once, it was due to binding in the drive train and when the driver moved the joystick forward and reverse rapidly enough, the current draw on drive motors caused a drop in voltage significant enough to cause the crio to reboot. The event log will show if voltage drops are causing this.

Canon reeves 19-03-2014 10:06

Re: Possible FMS problems.
 
Quote:

Originally Posted by Al Skierkiewicz (Post 1361222)
Canon,
The issue you describe is usually attributed to one or more of the following problems in no particular order.
1. The cRio is bumping into robot frame or is attached to robot frame electrically.
2. Wire whisker is touching the opposite polarity on the power supply wiring at either the cRio end or the PD end.
3. Power wiring is not inserted into the connector properly, a tug on the wiring will show this problem.
4. Rare manufacturing defects have been observed on the main breaker. while the robot is thruned on and booted, trying lightly tapping the red reset button on the breaker. If the lights on the robot blink, replace the main breaker.

Yesterday we tested for all of these, we replaced all the connectors from the CRIO to the PD board, we tried shaking the CRIO to get it to reboot, we also tried the tapping on the reset, and pulling on the wires. I think what we will have to do next is test the PD board connections to the motor controllers and all of the breakers. We also put in two new vector motor controllers this week and our mentor also had us wire the fans to a 30 amp slot on the PD board.

Canon reeves 19-03-2014 10:09

Re: Possible FMS problems.
 
Quote:

Originally Posted by noceradave (Post 1361255)
Check your event log from the match and see what happened:

http://wpilib.screenstepslive.com/s/...og-file-viewer

We had a similar problem once, it was due to binding in the drive train and when the driver moved the joystick forward and reverse rapidly enough, the current draw on drive motors caused a drop in voltage significant enough to cause the crio to reboot. The event log will show if voltage drops are causing this.

Will this still be accesible when we aren't connected to a field and still driving around? Thanks for the link, we will definitely use it!

Canon reeves 19-03-2014 10:11

Re: Possible FMS problems.
 
Quote:

Originally Posted by TheRamAlakazaam (Post 1361142)
Do you use a smart dashboard..
Our robot would reboot after autonomous because of it.
Turns out only Java Users can use it. So my team decided to set it to false along with other teams and we are all driving "blind" basically.

I'm not sure if we are or not, we are using the standard clamshell for driving, we have the same dashboard as last year and it was fine last year. I will ask the programmer later if we are but I doubt it.

geomapguy 19-03-2014 10:14

Re: Possible FMS problems.
 
Quote:

Originally Posted by Canon reeves (Post 1361259)
I'm not sure if we are or not, we are using the standard clamshell for driving, we have the same dashboard as last year and it was fine last year. I will ask the programmer later if we are but I doubt it.

Highly doubt you guys are using SmartDashboard since your team uses LabVIEW. I think I recall seeing the regular dashboard when playing with you guys. So i don't think what Raul said applies

DjScribbles 19-03-2014 10:22

Re: Possible FMS problems.
 
Quote:

Originally Posted by Canon reeves (Post 1361258)
Will this still be accesible when we aren't connected to a field and still driving around? Thanks for the link, we will definitely use it!

The event log is created by the driver station application local to your PC, and stores the logs on your hard drive. If you've been using the same PC for a few years, it will take a long time to load each time (until you go to the logging folder and delete old logs, use the windows sort by date to do this as log file name patterns changed year to year)

I only really discovered this utility at our last event (Southfield week 1) where the awesome FTA used it to help us diagnose some on field problems, it's very very helpful as it shows a graph of battery voltage, cpu/ram usage, and any errors that were sent. After getting comfortable with it, I was able to squash a few more issues from the viewing the logs.

G_rupp 19-03-2014 10:28

Re: Possible FMS problems.
 
Quote:

Originally Posted by Canon reeves (Post 1361252)
There has been a lot of mention of a frame short, could we test for this by testing the frame with a multimeter? we did that and it wasn't conducting anything, we checked the battery and all the connections to the pd board with it and got a consistent 12.5 volts. After the testing when the robot drove forward and shut off, we tried turning it back on, it took a few times but after it did turn back on we had no connection whatsoever, we tried rebooting, reset the radio, re-deployed code, exited the driver station then came back in. Under the diagnostic page on the driver station there was no light for the bridge, but there was a red light beside "robot". I've been trying to get it to shut off by shaking and dropping, and it still isn't rebooting like it is when I drive it. Is there anything else in the driver station we can use to diagnose the problem? Thank you guys so much, the team really appreciate's your help and GP!

Are you using any of the DIO connectors? I have seen instances where the +5V wire from the PWM cable will short intermittently to ground. This is normally seen when the White and Black wires are used for a Limit switch. The RED wire is cut and left hanging instead of being cut back and taped. The end of the wire will come in contact with the Black wire causing the 5VDC in the Digital Sidecar to turn off.

Al Skierkiewicz 19-03-2014 10:40

Re: Possible FMS problems.
 
Canon,
You can use a multimeter to check for continuity between the frame and each output terminal of every speed controller with the power off. You will likely find a short on one of them if this is related only to driving. It is normal for motors to have some resistance (2k-10k) to frame as the brush dust builds up inside. It is not normal to have a reading under 100 ohms. I am thinking you are going to find a shorted wire(s) in one of those outputs. There are two things that are known shutdown issues for the cRio. That is the cRio tied to the frame with another path somewhere, or a simple short on the output of a controller. If you are using Jags, the short should show up as a fault on the Jag that is affected even without motion I think.

Canon reeves 19-03-2014 11:26

Re: Possible FMS problems.
 
Quote:

Originally Posted by Al Skierkiewicz (Post 1361270)
Canon,
You can use a multimeter to check for continuity between the frame and each output terminal of every speed controller with the power off. You will likely find a short on one of them if this is related only to driving. It is normal for motors to have some resistance (2k-10k) to frame as the brush dust builds up inside. It is not normal to have a reading under 100 ohms. I am thinking you are going to find a shorted wire(s) in one of those outputs. There are two things that are known shutdown issues for the cRio. That is the cRio tied to the frame with another path somewhere, or a simple short on the output of a controller. If you are using Jags, the short should show up as a fault on the Jag that is affected even without motion I think.

So the output from the vector to the CIM motors? we are running 4 CIM motors as well. So we need to put the hot wire to the output and the ground on the multimeter to the frame and if we get a reading of 100 ohms we need to replace the motor controller? or is it the wire from the PD board to the Vector? Thank you so much for your help!

Al Skierkiewicz 19-03-2014 11:35

Re: Possible FMS problems.
 
Canon,
You need to check each Victor output screw to robot frame. The fans on the Victors are legal to be wired to the input terminals of the Victor to which they are mounted. They do not need to go back to the PD. Since you can only put one wire into a WAGO terminal, and it must be #18 or larger, wiring the fans to the PD is not practical. It is possible for metalic debris to enter a Victor in such a fashion that it will show a short in only one direction. However, that usually shows up in a very spectacular fashion.

Canon reeves 19-03-2014 12:54

Re: Possible FMS problems.
 
I tested the output screws on each side, on one side I got .3 resistance, on the other .4, is this ok? I am also getting a 12.5 volt reading on the input to the victors. I'm not sure how to test to the frame. I connected one part to the screw and the other to the frame and get zero everytime.

Al Skierkiewicz 19-03-2014 13:13

Re: Possible FMS problems.
 
Canon,
You put one meter probe on each of the output screws and one on a metallic part of the frame with the power off. You should read something higher than 2000 ohms. If you are reading the same value (~zero ohms) as when you touch the two probes together, then you have something shorted to the frame.

Canon reeves 19-03-2014 14:22

Re: Possible FMS problems.
 
Ok, thank you for sharing you wisdom with us! I am relatively new to FRC and wiring in general, so if these questions seem very basic please bear with me. Why do we want to have 2000+ ohms from the frame to the vector output? Why does the short happen when it isn't running through the frame? You do mean the drivetrain frame correct? here is the link to the multimeter we have, https://pioneers.berkeley.edu/wiki/H...e_a_Multimeter from what you have said we do need to set it to the ohmmeter setting correct? So when we discover the short, what would be the proper steps to fix the problem and ensure it doesn't happen again?

RobotKnight2014 19-03-2014 14:25

Re: Possible FMS problems.
 
Quote:

Originally Posted by Richard Wallace (Post 1361201)
Our team is in Michigan -- the issue I mentioned occurred at the Southfield District in Week 1. The intermittent chassis short was caused by some wiring that had previously provided power to a custom circuit (LED string), which had been removed after it was damaged during a practice match.

What was the issue seen at Utah?

There was a team at Utah who lost communication sporadically during teleop, the mentor who I saw talking to the FTA about it looked like your profile picture. I also thought there was a team called the average Joes there, must have been mistaken.

Al Skierkiewicz 19-03-2014 15:07

Re: Possible FMS problems.
 
Canon,
The short to the frame is an issue due to wiring errors, bare wire or lack of insulation. The 2000 ohms to frame is a normal condition with CIM motors that have been used for a while. The carbon dust worn from the brushes collects on the end bell of the motor inside and causes some leakage currents. When they get below 2000 ohms I start to worry. I speak to the drive train because you seemed to indicate the issue occurs only when driving. On the meter you show, if you set it to the position just below ohms, it will beep when you have a low resistance connection. If you find anything that beeps when one probe is connected to the frame, you have a serious problem.

Canon reeves 19-03-2014 18:30

Re: Possible FMS problems.
 
I do believe we located 1/2 of the problem. I did the tests with the multimeter and it was fine, I then turned on the robot and began checking the converter for the radio, when I did I heard this high pitched whining noise, and I noticed this happened when I tugged on the Wago for the converter on the PD board. I could cause the entire robot to briefly shut off by slightly tugging on the connector. When I checked the connector it was charred black around the wire, and when I pulled the wire out it was as if all of the individual strands had been saughtered together. I replaced the wago and pulled on it and it didn't shut off. We then tried to run the robot, but got no communication or code, but it did when we tethered it to the clamshell. I then drove back and forth and when I went forward then jerked back the entire robot shut off, the main breaker had been tripped. I pushed the red button on it then tried to turn it back on and it wouldn't turn on then I had to leave. It did this as well yesterday, and ever since then we hadn't gotten any communication or code unless we are tethered? Any probable causes?

geomapguy 19-03-2014 18:51

Re: Possible FMS problems.
 
Quote:

Originally Posted by Canon reeves (Post 1361490)
I do believe we located 1/2 of the problem. I did the tests with the multimeter and it was fine, I then turned on the robot and began checking the converter for the radio, when I did I heard this high pitched whining noise, and I noticed this happened when I tugged on the Wago for the converter on the PD board. I could cause the entire robot to briefly shut off by slightly tugging on the connector. When I checked the connector it was charred black around the wire, and when I pulled the wire out it was as if all of the individual strands had been saughtered together. I replaced the wago and pulled on it and it didn't shut off. We then tried to run the robot, but got no communication or code, but it did when we tethered it to the clamshell. I then drove back and forth and when I went forward then jerked back the entire robot shut off, the main breaker had been tripped. I pushed the red button on it then tried to turn it back on and it wouldn't turn on then I had to leave. It did this as well yesterday, and ever since then we hadn't gotten any communication or code unless we are tethered? Any probable causes?

Has the radio been reset since competition??

Canon reeves 19-03-2014 18:53

Re: Possible FMS problems.
 
Quote:

Originally Posted by geomapguy (Post 1361502)
Has the radio been reset since competition??

Yes, I reset it and made sure it as in ap 2.

Al Skierkiewicz 23-03-2014 11:09

Re: Possible FMS problems.
 
Canon,
What you describe is typical when the wire strands come out of the WAGO and contact the other polarity. What you are doing is shorting out the power supply on the PD. I would check that you have wired the WAGO correctly when you replaced it. If that is not an issue, then you may have killed the PD power supply. The only repair is a replacement.


All times are GMT -5. The time now is 07:45.

Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi