Go to Post the hardest task this year is the same as every other year: getting the engineers out of the crate so you can close it up and the FedEx guy can haul it away! - KenWittlief [more]
Home
Go Back   Chief Delphi > Technical > Control System > FRC Control System
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 04-30-2018, 08:27 AM
Coalman321's Avatar
Coalman321 Coalman321 is offline
Registered User
FRC #4145 (Worbots)
Team Role: Programmer
 
Join Date: Nov 2016
Rookie Year: 2015
Location: Central Ohio
Posts: 15
Coalman321 is an unknown quantity at this point
Exclamation System wide stutter in auton after FMS crash on Carson

My team, 4145, Was on the field during the 11th set of practice matches on Carson. About 80ish seconds into the match, the field hard crashed (along with four other fields) and the robots ground to a halt. About 2 minutes later, the field resumed for six seconds then gave the field fault noise. We were told to go reset them to the starting position and the match would be replayed. We powered our robot off and pushed it back. After 30 seconds the robot was powered back on and we walked off the field. The first run, our auton worked amazingly (got the first cube, missed the second) but when the second run of the match started, the whole robot -- shooter motors, drive train and lift -- all stuttered over two intervals of about a second and a half each. This really messed with our auto and caused the system to misfire when the switch was nowhere nearby. We tried power cycling, redeploying code, and finally outright reformatting the RIO. this was the only one that worked to fix the problem. We have our system logs from the three times it took us to fix the problem and the all show the same pattern with normal CANBus and CPU usage on the RIO. I am wondering what has access to the generation of ALL I/O, could cause the robot to rapidly enable / disable and can be manipulated by the Driver station / FMS? Personally my bet is on a system watchdog that got stuck somewhere.


DS LOG IMAGE: https://drive.google.com/file/d/1cHr...ew?usp=sharing
PRE-ISSUE: https://drive.google.com/file/d/1Nqx...ew?usp=sharing
ISSUE: https://drive.google.com/file/d/1AYe...ew?usp=sharing

(The videos may not be up yet)
Reply With Quote
  #2   Spotlight this post!  
Unread 04-30-2018, 09:58 AM
770352 770352 is offline
Registered User
AKA: Harold Griswold
FRC #3572 (Wavelength)
Team Role: Student
 
Join Date: Oct 2016
Rookie Year: 2016
Location: Muskegon, Michigan, United States
Posts: 21
770352 is an unknown quantity at this point
Re: System wide stutter in auton after FMS crash on Carson

My guess would be a dead battery, Especially if it was after a replay. Unless you rely on Encoders.
Reply With Quote
  #3   Spotlight this post!  
Unread 04-30-2018, 10:16 AM
Coalman321's Avatar
Coalman321 Coalman321 is offline
Registered User
FRC #4145 (Worbots)
Team Role: Programmer
 
Join Date: Nov 2016
Rookie Year: 2015
Location: Central Ohio
Posts: 15
Coalman321 is an unknown quantity at this point
Re: System wide stutter in auton after FMS crash on Carson

We run Pure pursuit and talon SRX velocity PID's. I hope we have encoders...
In all seriousness though, we get loads of telemetry back from the robot from encoder distances, velocities, field position, voltages and the like. All of this data points to a system wide and non-code problem. Keep in mind, no code was changed between matches only an FMS crash separated the two. The DS logs confirm this is not memory or CPU usage causing the problem on the RIO.
Reply With Quote
  #4   Spotlight this post!  
Unread 04-30-2018, 12:49 PM
Greg McKaskle Greg McKaskle is offline
Registered User
FRC #2468 (Team NI & Appreciate)
 
Join Date: Apr 2008
Rookie Year: 2008
Location: Austin, TX
Posts: 4,861
Greg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond repute
Re: System wide stutter in auton after FMS crash on Carson

It would be useful to see the DS log file of the auto that worked.

The voltage drop as the robot accelerates looks like the motors are being pulsed rather than run at a relatively constant speed.

The roboRIO CPU is actually quite high during this match, and if accompanied by lots of print statements, it may be interfering with motor controller enabling.

Are you using CAN for motor controllers?
Greg McKaskle
Reply With Quote
  #5   Spotlight this post!  
Unread 04-30-2018, 12:58 PM
Coalman321's Avatar
Coalman321 Coalman321 is offline
Registered User
FRC #4145 (Worbots)
Team Role: Programmer
 
Join Date: Nov 2016
Rookie Year: 2015
Location: Central Ohio
Posts: 15
Coalman321 is an unknown quantity at this point
Re: System wide stutter in auton after FMS crash on Carson

Hopefully I can procure the successful log today. As to the controllers, our system is mixed with the drivetrain being can and the intake and fork angle motors being pwm. Also the cpu usage is nominal for us. We have one 10ms loop and a whole bunch of 20ms loops as well for other subsystems on the bot. Thanks for the help!
Reply With Quote
  #6   Spotlight this post!  
Unread 04-30-2018, 01:03 PM
Coalman321's Avatar
Coalman321 Coalman321 is offline
Registered User
FRC #4145 (Worbots)
Team Role: Programmer
 
Join Date: Nov 2016
Rookie Year: 2015
Location: Central Ohio
Posts: 15
Coalman321 is an unknown quantity at this point
Re: System wide stutter in auton after FMS crash on Carson

Errr scratch that 2 10ms loops.
Reply With Quote
  #7   Spotlight this post!  
Unread 04-30-2018, 09:46 PM
Coalman321's Avatar
Coalman321 Coalman321 is offline
Registered User
FRC #4145 (Worbots)
Team Role: Programmer
 
Join Date: Nov 2016
Rookie Year: 2015
Location: Central Ohio
Posts: 15
Coalman321 is an unknown quantity at this point
Re: System wide stutter in auton after FMS crash on Carson

So I went and got a hold of the DS logs themselves from the first and second part of the practice match that crashed.
They don't show anything outstanding other than the voltage differences when the system was spiking the motors by enabling / disabling. Even the event log for the most part is useless.

DS log files here:github link
Reply With Quote
  #8   Spotlight this post!  
Unread 05-01-2018, 06:04 AM
AustinShalit's Avatar
AustinShalit AustinShalit is offline
Registered User
AKA: אוסטין
no team (WPILib Suite Developer)
 
Join Date: Dec 2013
Rookie Year: 2008
Location: Los Angeles/Worcester/Israel
Posts: 232
AustinShalit has much to be proud ofAustinShalit has much to be proud ofAustinShalit has much to be proud ofAustinShalit has much to be proud ofAustinShalit has much to be proud ofAustinShalit has much to be proud ofAustinShalit has much to be proud ofAustinShalit has much to be proud ofAustinShalit has much to be proud of
Re: System wide stutter in auton after FMS crash on Carson

Repo is private & please post robot code.
__________________
Reply With Quote
  #9   Spotlight this post!  
Unread 05-01-2018, 08:27 AM
Coalman321's Avatar
Coalman321 Coalman321 is offline
Registered User
FRC #4145 (Worbots)
Team Role: Programmer
 
Join Date: Nov 2016
Rookie Year: 2015
Location: Central Ohio
Posts: 15
Coalman321 is an unknown quantity at this point
Re: System wide stutter in auton after FMS crash on Carson

Working on the privacy settings now hopefully will fixed by the end of today. Unfortunately I do not control the repository settings but here is the code link when it works.

Code Repo:2018 Java Robot
Reply With Quote
  #10   Spotlight this post!  
Unread 05-01-2018, 08:46 AM
Bryce2471's Avatar
Bryce2471 Bryce2471 is offline
Alumnus
AKA: Bryce Croucher
FRC #2471 (Team Mean Machine)
Team Role: Mechanical
 
Join Date: Feb 2013
Rookie Year: 2006
Location: Camas, WA
Posts: 652
Bryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond repute
Re: System wide stutter in auton after FMS crash on Carson

Quote:
Originally Posted by 770352 View Post
My guess would be a dead battery, Especially if it was after a replay. Unless you rely on Encoders.
I'm guessing not, considering that their resting voltage was 12.8ish.
It's weird that I'm not seeing amperage draw from the PDP, but maybe it was unchecked when you took the screenshot.

Your CPU usage is a tad high, but I doubt it would cause a problem unless it was peged at 99%.

My guess is definitely a field issue. Notice that the field didn't randomize the plates before starting the match. Very odd behavior.
__________________
2006-2011:
FLL Team Future imagineers
2011-????:
FRC Team Mean Machine
Reply With Quote
  #11   Spotlight this post!  
Unread 05-01-2018, 09:28 AM
Maxcr1's Avatar
Maxcr1 Maxcr1 is offline
Registered User
AKA: Max D "Code Boy"
FRC #2169 (KING TeC)
Team Role: Programmer
 
Join Date: Nov 2016
Rookie Year: 2016
Location: Savage, MN
Posts: 113
Maxcr1 is a splendid one to beholdMaxcr1 is a splendid one to beholdMaxcr1 is a splendid one to beholdMaxcr1 is a splendid one to beholdMaxcr1 is a splendid one to beholdMaxcr1 is a splendid one to beholdMaxcr1 is a splendid one to behold
Re: System wide stutter in auton after FMS crash on Carson

Quote:
Originally Posted by Bryce2471 View Post
I'm guessing not, considering that their resting voltage was 12.8ish.
It's weird that I'm not seeing amperage draw from the PDP, but maybe it was unchecked when you took the screenshot.

Your CPU usage is a tad high, but I doubt it would cause a problem unless it was peged at 99%.

My guess is definitely a field issue. Notice that the field didn't randomize the plates before starting the match. Very odd behavior.
That's a good point. But it doesn't look like anybody else stuttered, which makes that unlikely. My guess is that it has something to do with the short periods of dead air from the FMS that can be seen by the disruptions in the top green line on your DS log, as their timing seems to match perfectly to the stutters. My guess is that something went wrong with the communication between your radio and the FMS (very possibly packet collision, used to happen all the time on DLINKs), causing your robot to lose connection to the FMS for a moment and go into disabled mode for a few loops, causing system-wide stuttering. If you haven't solved it yet, I recommend reflashing your radio. Often this will clear any collision-bound settings/channels from the system and will resolve your connection issues with the field. I know that we always keep an extra flashed radio on our Robot Cart just in case, and I recommend you do the same.
__________________
Max DeVos | Programmer | KING TeC 2169

Reply With Quote
  #12   Spotlight this post!  
Unread 05-01-2018, 09:35 AM
MrRoboSteve MrRoboSteve is offline
Mentor
AKA: Steve Peterson
FRC #3081 (Kennedy RoboEagles)
Team Role: Mentor
 
Join Date: Mar 2012
Rookie Year: 2011
Location: Bloomington, MN
Posts: 764
MrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond repute
Re: System wide stutter in auton after FMS crash on Carson

Quote:
Originally Posted by Maxcr1 View Post
My guess is that something went wrong with the communication between your radio and the FMS (very possibly packet collision, used to happen all the time on DLINKs), causing your robot to lose connection to the FMS for a moment and go into disabled mode for a few loops, causing system-wide stuttering.
While not perfect, the latency and packet loss shown in the posted log don't seem to support your hypothesis. It would be interesting to see logs from successful matches to compare
__________________
2018 UofM Kickoff, Great Northern Regional, Midwest Regional, Minneapolis regionals, Daly@CMP, Minnesota State Tournament, who knows how many offseason events, but won't be playing Santa at Minne Mini this year
2011 - present FRC 3081 Kennedy RoboEagles mentor
2013 - present event volunteer at[list truncated]
http://twitter.com/MrRoboSteve www.linkedin.com/in/speterson Microsoft is hiring
Reply With Quote
  #13   Spotlight this post!  
Unread 05-01-2018, 10:24 AM
Coalman321's Avatar
Coalman321 Coalman321 is offline
Registered User
FRC #4145 (Worbots)
Team Role: Programmer
 
Join Date: Nov 2016
Rookie Year: 2015
Location: Central Ohio
Posts: 15
Coalman321 is an unknown quantity at this point
Re: System wide stutter in auton after FMS crash on Carson

Just my luck... The repo wont go public until tonight probably. I do have the logs in full res this time though, good and bad. hope this helps.

Screen Captures:
bad
good
Reply With Quote
  #14   Spotlight this post!  
Unread 05-01-2018, 10:27 AM
Coalman321's Avatar
Coalman321 Coalman321 is offline
Registered User
FRC #4145 (Worbots)
Team Role: Programmer
 
Join Date: Nov 2016
Rookie Year: 2015
Location: Central Ohio
Posts: 15
Coalman321 is an unknown quantity at this point
Re: System wide stutter in auton after FMS crash on Carson

Quote:
Originally Posted by Maxcr1 View Post
If you haven't solved it yet, I recommend reflashing your radio.
As stated initially, we tried power cycling, redeploying code, and finally outright reformatting the RIO. This was the only one that worked to fix the problem.
Reply With Quote
  #15   Spotlight this post!  
Unread 05-01-2018, 12:37 PM
Coalman321's Avatar
Coalman321 Coalman321 is offline
Registered User
FRC #4145 (Worbots)
Team Role: Programmer
 
Join Date: Nov 2016
Rookie Year: 2015
Location: Central Ohio
Posts: 15
Coalman321 is an unknown quantity at this point
Re: System wide stutter in auton after FMS crash on Carson

Quote:
Originally Posted by Coalman321 View Post
Working on the privacy settings now hopefully will fixed by the end of today. Unfortunately I do not control the repository settings but here is the code link when it works.
The link should now Work as the repo is no longer private. The driver station logs are now stored in the RIObug directory off of root.

Code Link:https://github.com/Worthington-Robot...team4145/robot

Rio logs link:https://github.com/Worthington-Robot...ot_Test/RIObug
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 09:32 AM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi