OCCRA
Go to Post This is proof we can change the world. - Al Skierkiewicz [more]
Home
Go Back   Chief Delphi > Technical > Technical Discussion
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 03-04-2018, 10:25 AM
minisolarclown's Avatar
minisolarclown minisolarclown is offline
Registered User
FRC #3603 (Cyber Coyotes)
Team Role: Programmer
 
Join Date: Dec 2016
Rookie Year: 2015
Location: Reed City
Posts: 81
minisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really nice
Exclamation I need help with diagnosing a communications issue.

I apologize in advance for the length of this explanation of our issues and troubleshooting steps, but I felt every bit of information mattered. Feel free to ask me to expand on any info/clarify. Our issue is that we would randomly lose communication in random matches at our week 1 competition. We have taken several troubleshooting steps with no luck. I'm asking here to see if anyone can provide more advice or try and point us in the right direction.

The issue caused us to be disabled for half of our matches during week 1. Our robot did fantastic when it was enabled, though. Even though we could only participate half of the time, we ranked 13 in the qualifications and were chosen by alliance 3 for eliminations. Anyway, back to the issue.

We never had a single issue at home with it. I was able to run and test all of our autons without a hitch, as well as do teleop practice. Our first three matches on Friday went perfectly. But then after that, almost every single match our robot disconnected from the driver station. We had not made any electrical or programming changes leading up to that point. After three matches of losing communications for the entire match, we weren't even able to communicate with the robot to test in the pits. We then had an FTA member assigned to our team until the problem was solved.

The FTA guy (Alex) had us change our radio to a loaner one that had no issues, but the problem persisted, so it wasn't the radio.

While we were in queue for a match, we delayed the matches to re-image the roborio to see if that was the problem. 15 minutes later, we were on the field and everything ran smoothly for two more matches. But then the problem came back. When it came back the first time after the re-image, auton was able to run to completion, but it lost connection twice during teleop.

Alex went through the rio logs and discovered we would randomly brownout, causing us to lose communication. Remember how I said we never changed anything electrical or programming wise since when it was working at home? That's the weird part. Also in the riologs I can occasionally see brownouts that do not result in a loss of communication.

A consistent problem that we knew about was if you push the wires of the battery a certain way, the robot would lose power. We knew about this issue for about 4 weeks and hadn't been able to find the cause. Yesterday at about noon we found that our battery connector on the robot had a defective terminal. We replaced the terminal, but we would still randomly lose communication.

Our driver station does not have an ethernet port, so we use a USB to ethernet adapter. After a while, we would plug in to the roborio directly via USB and not have communications. My programming laptop and Alex's laptop were both able to connect. Upon further inspection, we discovered that the UDB port on the driver station was gradually breaking more and more throughout the match. So of course, the obvious fix is to use a different computer with a functioning USB port. We went out to our next match and we still lost communication.

Throughout the competition days, the communication loss was random, so about 50% of our matches we were able to run through without an issue. When we had communications, our robot performed amazingly. Despite not competing half of the time, we ended up finishing the qualifiers with rank 13. We were chosen by alliance 3 as their first pick, and we accepted. The first elimination match we had communication the entire time, but we still lost. We had some time to do some diagnosing before our second match. Again, we had total communication loss when plugged in directly. Only Alex's computer could detect it, which we discovered when we were in queue. The good news was a team had grabbed onto the scale plate and bent it ~30 degrees down, so we had plenty of time to troubleshoot.

We tried re-imaging the roborio again. It did not work, so we tried again. This time it was successful. We also noticed in the web-interface that the firmware version was ~3, so we brought it up to version 5. We thought everything was all set. The volunteers had finished fixing the scale, so we went to play in our second elimination match. We lost communication in the very beginning, ~50 seconds later we had connection again for about half a second, and then was out for the rest of the match. We lost 363-35, so we were eliminated. When the FTA staff went out to our robot once the match ended, they noticed the lift motor was pretty warm, even though we weren't enabled the entire match. It wouldn't have been warm from sometime beforehand because beforehand we were re-imaging it twice

Alex asked the other FTA staff members if we could use the field after the awards ceremony to do some testing. We were allowed to. However, when the time came for us to do our testing, there was another robot with a problem that kept the FMS from being able to start the practice match, so we had to go home because they weren't able to enable us.

I took the driver station and the programming laptop home last night so I could look at the rio logs for information, which we already checked at the competition. Sometimes we lose communications from brownouts, sometimes it happens without any reason that the riologs can show me. Other times I can see a brownout indicator without a loss of communication. Again, the robot was working fine when we were at home, and again we never changed anything.

I asked my mentor last night when we made it back to the school if he was able to contact FIRST to see if we can get extra unbag time strictly for troubleshooting. He said they probably wouldn't, but it's worth a shot. Our next competition is week 6. If on the off chance that we do get more unbag time to fix this issue, Alex says he can meet up with us to fix it.

Any help is greatly appreciated. If this problem doesn't get fixed before our next competition, we don't stand a chance of progressing.
Reply With Quote
  #2   Spotlight this post!  
Unread 03-04-2018, 02:20 PM
IKE's Avatar
IKE IKE is offline
Not so Custom User Title
AKA: Isaac Rife
no team (N/A)
Team Role: Mechanical
 
Join Date: Jan 2008
Rookie Year: 2003
Location: Michigan
Posts: 2,287
IKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond repute
Re: I need help with diagnosing a communications issue.

The brownouts have me a bit worried. When testing at home, were you testing on carpet? Tile will tend to draw much lighter current as it is slicker and requires less torque/current to drive around.

I assume you have done this, but trace the poer path from battery to PDP (high current traces) giving a strong wiggle. Loose connections there can cause voltage loss when higher current pulls happen.

Then verify on PDP that the fuses are fully inserted. The robot will function with the fuses lightly paced, but bumps and bounces can cause the fuses to momentarily dislodge if they are lightly put in.

Now check the power path from the PDP to the roborio. verify connections are proper configuration and secure.

Verify the PDP to VRM to Radio. Again proper configuration and secure.

*********
What style of drivetrain do you have? What ground speed are you geared to? What other power draw elements or big motors do you have? Too high of motor current draw can cause voltage dips which in turn cause the brownouts. Generally the brownout is designed to keep COMMs funtional, but I believe a high enough draw might be able to cause some sort of reset.

When you loose COMMs, does it come back after a while, or only after a full resset for the next match?

Have you verified there are no shorts to the chassis/frame?

Are you monitoring/recharging your batteries?
Reply With Quote
  #3   Spotlight this post!  
Unread 03-04-2018, 02:41 PM
minisolarclown's Avatar
minisolarclown minisolarclown is offline
Registered User
FRC #3603 (Cyber Coyotes)
Team Role: Programmer
 
Join Date: Dec 2016
Rookie Year: 2015
Location: Reed City
Posts: 81
minisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really nice
Re: I need help with diagnosing a communications issue.

Thank you for your response.

Yes, at home we do test on carpet. Alex did check with a multimeter for continuity, but when we have unbag time I will do the same with wiggling the connections. We will also check the fuses along with the short to the frame.

We have a 6-wheel drive train. We are using the 3 CIM Ball Shifter by Vex. We did have to take the air compressor out because of weight, so we are only using the low gear. We have a CIM for our lift, and a CIM for an arm on the lift to raise and lower it. We have two mini-CIMs at the end of our arms for our intake. When the brownouts occur, the two intake motors aren't running.

When we lose COMMs, it usually goes away for about 30 seconds, and reconnects. Sometimes we can continue for the rest of the match, other times we lose COMMs immediately after we re-gain it.

We have 6 batteries, each with their own charger. After every match we grab the next in line, so we always have a full battery.
Reply With Quote
  #4   Spotlight this post!  
Unread 03-04-2018, 03:14 PM
tkchan tkchan is offline
Registered User
AKA: Joe
FRC #4546 (Shockwave)
Team Role: Mentor
 
Join Date: Jan 2016
Rookie Year: 2015
Location: Dover, NH
Posts: 7
tkchan is an unknown quantity at this point
Re: I need help with diagnosing a communications issue.

How old are your batteries? I assumed that you have the same issues regardless of WHICH battery you are using? or you trend to have more problem with specific battery?

I would suggested looking through your drive station log files using the DS_LogFileViewer and look at the overall power situation - current of all the major consumers (i.e. motors) as well as the voltage. Specifically look and see how much the voltage dips down to (during heavier current draw) and how much it comes back up when less current draw.
Reply With Quote
  #5   Spotlight this post!  
Unread 03-04-2018, 04:26 PM
minisolarclown's Avatar
minisolarclown minisolarclown is offline
Registered User
FRC #3603 (Cyber Coyotes)
Team Role: Programmer
 
Join Date: Dec 2016
Rookie Year: 2015
Location: Reed City
Posts: 81
minisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really nice
Re: I need help with diagnosing a communications issue.

All 6 batteries are brand new for this year. My mentor believes that battery 3 and 4 work better, but I haven't seen a quantitative difference in the performance of any of the batteries.

Looking at the logs, when the robot is enabled but not doing anything, the voltage is between 12.5 and 13.5. During teleop the largest voltage drop in any particular match goes to about 9.5-10 volts, but instantly jumps back up to about 11.5-12 volts. In matches where the brownouts occur, it doesn't actually show a drop in voltage.
Reply With Quote
  #6   Spotlight this post!  
Unread 03-04-2018, 04:46 PM
EricH's Avatar
EricH EricH is offline
New year, new team
FRC #1197 (Torbots)
Team Role: Engineer
 
Join Date: Jan 2005
Rookie Year: 2003
Location: SoCal
Posts: 21,910
EricH has a reputation beyond reputeEricH has a reputation beyond reputeEricH has a reputation beyond reputeEricH has a reputation beyond reputeEricH has a reputation beyond reputeEricH has a reputation beyond reputeEricH has a reputation beyond reputeEricH has a reputation beyond reputeEricH has a reputation beyond reputeEricH has a reputation beyond reputeEricH has a reputation beyond repute
Re: I need help with diagnosing a communications issue.

The lift motors being warm without being used is ringing all kinds of alarm bells in my head, even with them being MiniCIMs.

Could you, as soon as you are out of the bag for your next event, check for your lift motor wiring for shorts? They'd likely be intermittent shorts, but might be worth investigating.
__________________
Past teams:
2003-2007: FRC0330 BeachBots; 2008: FRC1135 Shmoebotics; 2012: FRC4046 Schroedinger's Dragons

"Rockets are tricky..."--Elon Musk



Reply With Quote
  #7   Spotlight this post!  
Unread 03-04-2018, 05:15 PM
minisolarclown's Avatar
minisolarclown minisolarclown is offline
Registered User
FRC #3603 (Cyber Coyotes)
Team Role: Programmer
 
Join Date: Dec 2016
Rookie Year: 2015
Location: Reed City
Posts: 81
minisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really nice
Re: I need help with diagnosing a communications issue.

Again, we do plan on asking FIRST if it's possible to have extra unbag time to diagnose a problem that no one can figure out. I'd like to think that they'd be a little leanient, because if we can't fix it, we've wasted thousands of dollars on a robot to not be able to compete. I don't believe the standard 6 hours of unbag time will be nearly enough to get it worked out.

The lift motor isn't a miniCIM, the intake motors are. The lift motor is just a single CIM with some gearing and an encoder. I do know that the motor gets warm when being used constantly, like in practice, but that's to be expected.

We only checked the motor temperature after our last elimination match, where we were only enabled for a grand total of about 3 seconds throughout the entire match. The FTA crew wanted to check our robot immediately after the match ended, so when it was still on the field. One of them felt the motor and noticed it was warm for being disabled basically the entire match.
Reply With Quote
  #8   Spotlight this post!  
Unread 03-04-2018, 06:43 PM
Pi Fighter's Avatar
Pi Fighter Pi Fighter is offline
Insert Monty Python Reference Here
AKA: Trevor
FRC #1718 (Fighting Pi)
Team Role: Mentor
 
Join Date: May 2009
Rookie Year: 2009
Location: Armada, MI
Posts: 58
Pi Fighter has a spectacular aura aboutPi Fighter has a spectacular aura aboutPi Fighter has a spectacular aura about
Re: I need help with diagnosing a communications issue.

Quote:
Originally Posted by minisolarclown View Post
Our driver station does not have an ethernet port, so we use a USB to ethernet adapter. After a while, we would plug in to the roborio directly via USB and not have communications. My programming laptop and Alex's laptop were both able to connect. Upon further inspection, we discovered that the UDB port on the driver station was gradually breaking more and more throughout the match. So of course, the obvious fix is to use a different computer with a functioning USB port. We went out to our next match and we still lost communication.
Do you know if the power management for the USB ports was modified? Most modern laptops have a setting that allows the power to the USB ports to be shut down to save battery.

To turn it off, go to Control Panel >> Hardware and Sound >> Device Manager >> Universal Serial Bus Controllers
Then, right click each USB Hub and select Properties >> Power Management >> Uncheck 'Allow the computer to turn off this device to save power.'

That may help with connection issues during the match, especially if you're using an Ethernet-to-USB adapter.
Reply With Quote
  #9   Spotlight this post!  
Unread 03-04-2018, 07:28 PM
minisolarclown's Avatar
minisolarclown minisolarclown is offline
Registered User
FRC #3603 (Cyber Coyotes)
Team Role: Programmer
 
Join Date: Dec 2016
Rookie Year: 2015
Location: Reed City
Posts: 81
minisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really nice
Re: I need help with diagnosing a communications issue.

Done, thank you for suggesting that. We'll know how it helps at our next competition. However, it isn't the only cause. Our dedicated driver station needs USB-> Ethernet, but the other laptop that we used instead for the second half of the competition has a built in Ethernet port.
Reply With Quote
  #10   Spotlight this post!  
Unread 03-04-2018, 08:05 PM
Tom Line's Avatar
Tom Line Tom Line is offline
Don't lay blame. Fix probems.
FRC #1718 (The Fighting Pi)
Team Role: Mentor
 
Join Date: Jan 2007
Rookie Year: 1999
Location: Armada, Michigan
Posts: 3,104
Tom Line has a reputation beyond reputeTom Line has a reputation beyond reputeTom Line has a reputation beyond reputeTom Line has a reputation beyond reputeTom Line has a reputation beyond reputeTom Line has a reputation beyond reputeTom Line has a reputation beyond reputeTom Line has a reputation beyond reputeTom Line has a reputation beyond reputeTom Line has a reputation beyond reputeTom Line has a reputation beyond repute
Re: I need help with diagnosing a communications issue.

Quote:
Originally Posted by minisolarclown View Post
All 6 batteries are brand new for this year. My mentor believes that battery 3 and 4 work better, but I haven't seen a quantitative difference in the performance of any of the batteries.

Looking at the logs, when the robot is enabled but not doing anything, the voltage is between 12.5 and 13.5. During teleop the largest voltage drop in any particular match goes to about 9.5-10 volts, but instantly jumps back up to about 11.5-12 volts. In matches where the brownouts occur, it doesn't actually show a drop in voltage.
This is an excellent clue. You would be seeing a voltage drop prior to a brownout. In addition you can turn the pdp values on in the graph to see current draw.
Reply With Quote
  #11   Spotlight this post!  
Unread 03-04-2018, 08:12 PM
chapman1's Avatar
chapman1 chapman1 is offline
Registered User
FRC #3688 (Suttons Bay Norsemen Robotics)
Team Role: Mentor
 
Join Date: Mar 2011
Rookie Year: 2011
Location: Suttons Bay MI
Posts: 161
chapman1 has a reputation beyond reputechapman1 has a reputation beyond reputechapman1 has a reputation beyond reputechapman1 has a reputation beyond reputechapman1 has a reputation beyond reputechapman1 has a reputation beyond reputechapman1 has a reputation beyond reputechapman1 has a reputation beyond reputechapman1 has a reputation beyond reputechapman1 has a reputation beyond reputechapman1 has a reputation beyond repute
Re: I need help with diagnosing a communications issue.

Quote:
Originally Posted by minisolarclown View Post
We have two mini-CIMs at the end of our arms for our intake.
I was at your event, and although I had little time to stroll the pits, I always try to have a look at your team's stellar designs. Also, my memory has been proven to be fallible on several occasions. With those caveats, I seem to remember 775-pros on your intake, and wires feeding them that seemed abnormally thin. It's quite possible I am wrong on the motors, and that your wires had really thin insulation. I'll risk being wrong and put it out there anyway, in an effort to help.
__________________
Older, but still somewhat coherent.
Reply With Quote
  #12   Spotlight this post!  
Unread 03-04-2018, 08:23 PM
minisolarclown's Avatar
minisolarclown minisolarclown is offline
Registered User
FRC #3603 (Cyber Coyotes)
Team Role: Programmer
 
Join Date: Dec 2016
Rookie Year: 2015
Location: Reed City
Posts: 81
minisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really nice
Re: I need help with diagnosing a communications issue.

Hey, anything helps! I'll check with the person who wired it so I can get some more information on it. Thank you for taking the time to check us out, it's good to hear that.
Reply With Quote
  #13   Spotlight this post!  
Unread 03-04-2018, 08:29 PM
Todd's Avatar
Todd Todd is offline
Software Engineer
FRC #1071 (Team Max)
Team Role: Mentor
 
Join Date: Feb 2005
Rookie Year: 2004
Location: Connecticut, Wolcott
Posts: 86
Todd has much to be proud ofTodd has much to be proud ofTodd has much to be proud ofTodd has much to be proud ofTodd has much to be proud ofTodd has much to be proud ofTodd has much to be proud ofTodd has much to be proud ofTodd has much to be proud of
Re: I need help with diagnosing a communications issue.

Can you upload some of your log files somewhere shareable?
__________________
Mike Todd
Team MAX 1071
Sikorsky Aircraft Corporation - Embedded Software Engineer
Guy that does stuff
Reply With Quote
  #14   Spotlight this post!  
Unread 03-04-2018, 09:10 PM
minisolarclown's Avatar
minisolarclown minisolarclown is offline
Registered User
FRC #3603 (Cyber Coyotes)
Team Role: Programmer
 
Join Date: Dec 2016
Rookie Year: 2015
Location: Reed City
Posts: 81
minisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really niceminisolarclown is just really nice
Re: I need help with diagnosing a communications issue.

I made a shared dropbox folder so you can view. I hand picked a few different scenarios because I didn't want you to have to pour over all of them to find specific ones. I chose some with matches that went well, there's a few testing ones, there's some with comm loss + brownout, comm loss only, and brownout only. Let me know if you would like more.
Reply With Quote
  #15   Spotlight this post!  
Unread 03-04-2018, 09:49 PM
Todd's Avatar
Todd Todd is offline
Software Engineer
FRC #1071 (Team Max)
Team Role: Mentor
 
Join Date: Feb 2005
Rookie Year: 2004
Location: Connecticut, Wolcott
Posts: 86
Todd has much to be proud ofTodd has much to be proud ofTodd has much to be proud ofTodd has much to be proud ofTodd has much to be proud ofTodd has much to be proud ofTodd has much to be proud ofTodd has much to be proud ofTodd has much to be proud of
Re: I need help with diagnosing a communications issue.

So to me this certainly looks electrical.

The signature of -most- of the failures seems to be a sudden loss in voltage - and subsequent activation of brownout protection on the rio - often not correlated with any actual current consumption.

It seems to be roughly 50/50 with the crio only losing power vs the radio also losing power. (you can tell from the ping messages in the event log)

I would double check the connections from your battery through to your rio and radio, some things to check:

Do you have a split ring washer on your PDP screw connections?
Are you using an appropriately long (10mm) M6 connector in the PDP?
Are you using nuts with lock washers on your main breaker?
Are the small rio/vrm fuses on your PDP in good shape and fully inserted into the PDP (it's very easy for these to come loose and still [mostly] function)

Do these failures in any way seem correlated with rough play on your robot? Collisions or the like?

This could certainly also be the result of a short - perhaps a frame short, have you ensured that no part of your frame is electrically connected?
Attached Thumbnails
Click image for larger version

Name:	rec 1.png
Views:	64
Size:	119.5 KB
ID:	23124  Click image for larger version

Name:	rec 2.png
Views:	40
Size:	104.9 KB
ID:	23125  Click image for larger version

Name:	rec 3.png
Views:	40
Size:	119.5 KB
ID:	23126  
__________________
Mike Todd
Team MAX 1071
Sikorsky Aircraft Corporation - Embedded Software Engineer
Guy that does stuff
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 02:50 AM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi