OCCRA
Go to Post PRY BAR: A tool used to crumple the metal surrounding that clip or bracket you needed to remove in order to replace a 50 cent part. - dlavery [more]
Home
Go Back   Chief Delphi > Technical > Programming
CD-Media  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 11-25-2018, 10:49 PM
dominikWin dominikWin is offline
Registered User
FRC #1014
Team Role: Alumni
 
Join Date: Oct 2018
Rookie Year: 2015
Location: Dublin, OH
Posts: 5
dominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of light
Badlog data logging framework

Hi everyone,

I wanted to share what I worked on this past season.
After an eventful 2017 year when Team 1014 first deployed a swerve drive bot, we spent months trying to iron out all the kinks. Logging every possible metric we had access to helped tremendously but resulted in some pretty hacked together code. It became clear that we needed a better system that would not only gather data reliably but also allow us to visualize it, and most importantly make it accessible to all team members. So, with memories of matching corrupt CSVs to match videos still fresh in mind I started working on a fix: badlog. In short, we get dynamically generated HTML versions of our data logs. Details and source code are available in the whitepaper.

So what makes badlog good?
Despite having a slight bit of a bias I can confidently say that using badlog as a user in the 2018 season made life easier as a programmer. We set up badlog once at the start of the season, and since then within seconds of every match we get intimidate visualized data. It never broke; it helped catch countless problems, big and small, at every competition; and is as risk-averse as you expect any logging system to be. All of these logs for the season are available online here; as they were during our competitions. Describing a visualization system with text doesn't do it justice, so that last link is a good example of the system.

Badlog has greatly helped our team by being a compelling alternative to conventional FRC logging. It provides useful information, as shown in the examples, and does this well, as shown in the whitepaper. Team 1014 is also making this available to other teams and is more than willing to provide assistance if needed, and with any luck team 1014 won't be the only ones showing off our logs this upcoming season.
Reply With Quote
  #2   Spotlight this post!  
Unread 11-26-2018, 01:57 AM
AllenGregoryIV's Avatar
AllenGregoryIV AllenGregoryIV is offline
Engineering Coach
AKA: Allen "JAG" Gregory
FRC #3847 (Spectrum)
Team Role: Coach
 
Join Date: Jul 2008
Rookie Year: 2003
Location: Texas
Posts: 3,071
AllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond repute
Send a message via AIM to AllenGregoryIV
Re: Badlog data logging framework

This is really cool, I'm excited to look into this more.
__________________

Team 647 | Cyber Wolf Corps | Alumni | 2003-2006 | Shoemaker HS
Team 2587 | DiscoBots | Mentor | 2008-2011 | Rice University / Houston Food Bank
Team 3847 | Spectrum | Coach | 2012-20... | St Agnes Academy
LRI | Alamo Regional | 2014-20...
"Competition has been shown to be useful up to a certain point and no further, but cooperation, which is the thing we must strive for today, begins where competition leaves off." - Franklin D. Roosevelt
Reply With Quote
  #3   Spotlight this post!  
Unread 11-26-2018, 06:23 AM
gerthworm's Avatar
gerthworm gerthworm is offline
Making the 1's and 0's
FRC #1736 (Robot Casserole)
Team Role: Mentor
 
Join Date: Jan 2015
Rookie Year: 2015
Location: Peoria, IL
Posts: 740
gerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond repute
Re: Badlog data logging framework

<3 for highcharts!!!

We've been doing time-series logging for the past couple years and it's repeatedly proved invaluable at finding root cause of issues, and even predicting issues before they happen!

For disk space issues, we log to a 16gb usb drive that we attache to the RIO. Has the added advantage of being able to pull it off the robot to download the log files, though we usually are just using a python script that does an scp operation.

Last edited by gerthworm : 11-26-2018 at 06:26 AM.
Reply With Quote
  #4   Spotlight this post!  
Unread 11-26-2018, 09:22 AM
Andrew Schreiber Andrew Schreiber is offline
Petitioning the UN to ban Lythgoe
no team
Team Role: Engineer
 
Join Date: Jan 2005
Rookie Year: 2000
Location: Somehow, the entire east coast...
Posts: 4,948
Andrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond repute
Re: Badlog data logging framework

Do you make use of the actual bag files or only the CSVs?

We're working on a similar approach but running into issues with bag files due to the size of the ones output from ROS. (even with compression we're in the multiple GB range)
__________________




.
Reply With Quote
  #5   Spotlight this post!  
Unread 11-26-2018, 09:38 AM
dominikWin dominikWin is offline
Registered User
FRC #1014
Team Role: Alumni
 
Join Date: Oct 2018
Rookie Year: 2015
Location: Dublin, OH
Posts: 5
dominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of light
Re: Badlog data logging framework

Quote:
Originally Posted by Andrew Schreiber View Post
Do you make use of the actual bag files or only the CSVs?
Just CSVs, but we don't call them CSVs in case someone tries to open them in excel and gets errors when it finds JSON in the header. Calling them bags is really just because other names were taken from ros and rosbags are the closest thing to this.
Reply With Quote
  #6   Spotlight this post!  
Unread 11-26-2018, 10:11 AM
dominikWin dominikWin is offline
Registered User
FRC #1014
Team Role: Alumni
 
Join Date: Oct 2018
Rookie Year: 2015
Location: Dublin, OH
Posts: 5
dominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of light
Re: Badlog data logging framework

As far as disk space goes we didn't really have any trouble. CSV with truncated decimals is really space efficient even without compression. We created files just under 2MB per match, so we could easily push 50+ matches without any worry. At one point had two separate competitions on the Rio at once and we had 147MB to spare.

So if you log around as much as we did (35-40 data topics) that ends up being around 4 hours to get to 150MB. Since we cut the rate down to 4 updates/s when disabled we can go around two full days on standby.
Reply With Quote
  #7   Spotlight this post!  
Unread 11-26-2018, 10:28 AM
Andrew Schreiber Andrew Schreiber is offline
Petitioning the UN to ban Lythgoe
no team
Team Role: Engineer
 
Join Date: Jan 2005
Rookie Year: 2000
Location: Somehow, the entire east coast...
Posts: 4,948
Andrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond repute
Re: Badlog data logging framework

So... they aren't actually ROS bag files at all just CSV files?
__________________




.
Reply With Quote
  #8   Spotlight this post!  
Unread 11-26-2018, 10:41 AM
dominikWin dominikWin is offline
Registered User
FRC #1014
Team Role: Alumni
 
Join Date: Oct 2018
Rookie Year: 2015
Location: Dublin, OH
Posts: 5
dominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of light
Re: Badlog data logging framework

Huh, looks like my last response got lost but a later one went through. Yeah they are all CSV files (with a JSON header), but not called that in case someone tries to open them in excel or something. Badlog takes a lot of its names from ros and since rosbag is the closest thing we ended up calling them bag files.
Reply With Quote
  #9   Spotlight this post!  
Unread 11-26-2018, 10:58 AM
gerthworm's Avatar
gerthworm gerthworm is offline
Making the 1's and 0's
FRC #1736 (Robot Casserole)
Team Role: Mentor
 
Join Date: Jan 2015
Rookie Year: 2015
Location: Peoria, IL
Posts: 740
gerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond repute
Re: Badlog data logging framework

Quote:
Originally Posted by dominikWin View Post
Badlog takes a lot of its names from ros and since rosbag is the closest thing we ended up calling them bag files.
If you do happen to decide to choose a new file extension in the future for your json+csv format, may I suggest ".badbag", purely because it's fun to say?
Reply With Quote
  #10   Spotlight this post!  
Unread 11-26-2018, 11:04 AM
dominikWin dominikWin is offline
Registered User
FRC #1014
Team Role: Alumni
 
Join Date: Oct 2018
Rookie Year: 2015
Location: Dublin, OH
Posts: 5
dominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of lightdominikWin is a glorious beacon of light
Re: Badlog data logging framework

Quote:
Originally Posted by gerthworm View Post
If you do happen to decide to choose a new file extension in the future for your json+csv format, may I suggest ".badbag", purely because it's fun to say?
Well badlog doesn't control what you name the files, that's left up to the user. I'm pretty sure you just settled what we'll be using next year though
Reply With Quote
  #11   Spotlight this post!  
Unread 11-26-2018, 11:21 AM
Andrew Schreiber Andrew Schreiber is offline
Petitioning the UN to ban Lythgoe
no team
Team Role: Engineer
 
Join Date: Jan 2005
Rookie Year: 2000
Location: Somehow, the entire east coast...
Posts: 4,948
Andrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond repute
Re: Badlog data logging framework

Quote:
Originally Posted by dominikWin View Post
Huh, looks like my last response got lost but a later one went through. Yeah they are all CSV files (with a JSON header), but not called that in case someone tries to open them in excel or something. Badlog takes a lot of its names from ros and since rosbag is the closest thing we ended up calling them bag files.
mmk, I was going to be VERY shocked if you'd flat out stollen the bag file format. Woulda made me happy since it would have just let us drop in our rosbag files
__________________




.
Reply With Quote
  #12   Spotlight this post!  
Unread 12-05-2018, 02:07 PM
coderkevin coderkevin is offline
Registered User
AKA: Kevin Killingsworth
FRC #2357 (System Meltdown)
Team Role: Coach
 
Join Date: Feb 2017
Rookie Year: 2012
Location: Peculiar, MO
Posts: 14
coderkevin is an unknown quantity at this point
Re: Badlog data logging framework

Thanks for posting this. I think we'll use some of our dwindling time before kickoff trying it out.
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 11:46 PM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin®
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi