OCCRA
Go to Post FIRST needs a disclaimer that parts of the field can become flying projectiles under hurricane force winds. - Barry Bonzack [more]
Home
Go Back   Chief Delphi > Technical > Control System
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 03-16-2012, 08:34 AM
iambujo iambujo is offline
Mentor
FRC #2729 (Storm)
Team Role: Mentor
 
Join Date: Feb 2008
Rookie Year: 2008
Location: NJ
Posts: 87
iambujo is a name known to alliambujo is a name known to alliambujo is a name known to alliambujo is a name known to alliambujo is a name known to alliambujo is a name known to all
Alternate viewer for Driver Station Logs (dslog)?

Hello - we are analyzing our ds logs due to lag issues seen at our first competition. I am wondering if the log structure/content is documented any where or if a portable simplified (windows or Linux) viewer has been identified. Ideally I'd like to view it in a way that I can create my own plots, crunch filter, etc without needed all the NI software installed. If methods exist already that would help!

Thanks
Reply With Quote
  #2   Spotlight this post!  
Unread 03-16-2012, 07:34 PM
Greg McKaskle Greg McKaskle is offline
Registered User
FRC #2468 (Team NI & Appreciate)
 
Join Date: Apr 2008
Rookie Year: 2008
Location: Austin, TX
Posts: 4,861
Greg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond repute
Re: Alternate viewer for Driver Station Logs (dslog)?

The DS log file is binary (for compactness and has a simple header of a 32 bit version number that is always zero at this point and a (64,64) bit timestamp with an epoch of 12:00:00.000 AM 1/1/1904 (Universal Time). It is a 64 bit int of seconds since ... and then 64 bits of fractional seconds.

The rest of the file is a sequence of tuples (single, single, single, u8, i8) and I believe they are packed with no alignment. The attached code build the plots and the legend shows the plots being built.

The only complicated element that the last six plots packed into the u8. Each bit selects between Nan (which lifts the graph pen), and a value plotted on the scale. It is basically a bitfield of those six bits representing what the DS told the robot to run and what the robot ran during that time slice.

I think the last element to add is that the byte ordering is big endian.

Let me know if I left anything out or something doesn't seem right.

Greg McKaskle
Attached Thumbnails
Click image for larger version

Name:	Screen Shot 2012-03-16 at 7.12.43 PM.png
Views:	169
Size:	16.9 KB
ID:	12332  Click image for larger version

Name:	Screen Shot 2012-03-16 at 7.15.00 PM.png
Views:	124
Size:	10.6 KB
ID:	12333  
Reply With Quote
  #3   Spotlight this post!  
Unread 03-16-2012, 08:43 PM
iambujo iambujo is offline
Mentor
FRC #2729 (Storm)
Team Role: Mentor
 
Join Date: Feb 2008
Rookie Year: 2008
Location: NJ
Posts: 87
iambujo is a name known to alliambujo is a name known to alliambujo is a name known to alliambujo is a name known to alliambujo is a name known to alliambujo is a name known to all
Re: Alternate viewer for Driver Station Logs (dslog)?

Wow, thanks for the reply Greg! I am not familiar with Labview so the image is a little 'greek' to me, but I am sure I can decipher it. If I have some free time (most likely post competition season at this point) I may write a little decoder app to expand the data and make it more portable for things like xplot and excel. Not sure if it will be useful, but I like a good challenge.
Reply With Quote
  #4   Spotlight this post!  
Unread 02-14-2014, 09:51 PM
fovea1959's Avatar
fovea1959 fovea1959 is offline
Herder of programmers
AKA: Doug Wegscheid
FRC #3620 (The Average Joes)
Team Role: Mentor
 
Join Date: Jan 2011
Rookie Year: 2011
Location: St Joseph
Posts: 385
fovea1959 will become famous soon enough
Re: Alternate viewer for Driver Station Logs (dslog)?

Greg: thanks for the information; I did a hex dump of some files tonight, your explanation makes sense...

it appears that after the initial time stamp, there is no timing information, and it's assumed that there is an assumed .020s between samples, regardless of reality? or are the samples written to the file on a .020s timer that doesn't have cumulative drift?
Reply With Quote
  #5   Spotlight this post!  
Unread 02-14-2014, 11:56 PM
Greg McKaskle Greg McKaskle is offline
Registered User
FRC #2468 (Team NI & Appreciate)
 
Join Date: Apr 2008
Rookie Year: 2008
Location: Austin, TX
Posts: 4,861
Greg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond repute
Re: Alternate viewer for Driver Station Logs (dslog)?

The samples are written in a timed loop that will attempt to keep on the 20ms schedule, no drift. It runs at a high priority and doesn't have much jitter. By the way, we are now at version 1. I believe I posted details earlier this week, but the only difference is that the mode bits shifted down.

Greg McKaskle
Reply With Quote
  #6   Spotlight this post!  
Unread 03-31-2014, 05:43 PM
Joe Ross's Avatar Unsung FIRST Hero
Joe Ross Joe Ross is offline
Registered User
FRC #0330 (Beachbots)
Team Role: Engineer
 
Join Date: Jun 2001
Rookie Year: 1997
Location: Los Angeles, CA
Posts: 8,984
Joe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond repute
Re: Alternate viewer for Driver Station Logs (dslog)?

Quote:
Originally Posted by Greg McKaskle View Post
The samples are written in a timed loop that will attempt to keep on the 20ms schedule, no drift. It runs at a high priority and doesn't have much jitter.
Is this the same timer that is sending data to the robot? Have you characterized the jitter as the DS CPU gets overloaded?
Reply With Quote
  #7   Spotlight this post!  
Unread 04-06-2014, 11:32 PM
Joe Ross's Avatar Unsung FIRST Hero
Joe Ross Joe Ross is offline
Registered User
FRC #0330 (Beachbots)
Team Role: Engineer
 
Join Date: Jun 2001
Rookie Year: 1997
Location: Los Angeles, CA
Posts: 8,984
Joe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond repute
Re: Alternate viewer for Driver Station Logs (dslog)?

Quote:
Originally Posted by Greg McKaskle View Post
By the way, we are now at version 1. I believe I posted details earlier this week, but the only difference is that the mode bits shifted down.
I couldn't find that description of the new format. Would you mind reposting it? Also, what is the format for the dsevents file?
Reply With Quote
  #8   Spotlight this post!  
Unread 04-09-2014, 06:25 AM
Greg McKaskle Greg McKaskle is offline
Registered User
FRC #2468 (Team NI & Appreciate)
 
Join Date: Apr 2008
Rookie Year: 2008
Location: Austin, TX
Posts: 4,861
Greg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond repute
Re: Alternate viewer for Driver Station Logs (dslog)?

The log file is not written out in the high priority timed loop. I try to keep that loop lean. It is done where we receive the status packets. What is logged is mostly the status info anyway. I have characterized the control packet loop with high CPU, but not the status or logging loop. The statistics on packet loss and latency are going to lag a bit just because of how they are measured and reported. The events tend to be even more lagged because they are often timeouts. But I have never seen a file where I thought the CPU usage messed up the data. If the DS doesn't have data for a time period, it replicates the new packet to fill in the void. So I would suspect that high packet loss would have flat spots for everything else and high bars for the packets.

The new format has version 1 and the trace bits are shifted down. Earlier in this thread, it shows a simple diagram that indexes the bits from the i8. It throws away the first two bits. We thought we were going to add more info and I modified code to prep. The result of that was that the bits are now packed starting at bit 0.

The dsevents file has the same header and then each event/warning is a timestamp and a LVString. The LVString is not NULL terminated, but is an i32 count of bytes followed by the bytes. The strings are possibly encoded a bit with <> tags, depending on who is sending the data to the DS.

Greg McKaskle
Attached Thumbnails
Click image for larger version

Name:	From Clipboard 6.png
Views:	89
Size:	99.8 KB
ID:	16797  
Reply With Quote
  #9   Spotlight this post!  
Unread 04-13-2014, 04:32 PM
Joe Ross's Avatar Unsung FIRST Hero
Joe Ross Joe Ross is offline
Registered User
FRC #0330 (Beachbots)
Team Role: Engineer
 
Join Date: Jun 2001
Rookie Year: 1997
Location: Los Angeles, CA
Posts: 8,984
Joe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond repute
Re: Alternate viewer for Driver Station Logs (dslog)?

Quote:
Originally Posted by Greg McKaskle View Post
The new format has version 1 and the trace bits are shifted down. Earlier in this thread, it shows a simple diagram that indexes the bits from the i8. It throws away the first two bits. We thought we were going to add more info and I modified code to prep. The result of that was that the bits are now packed starting at bit 0.
Looking at some data, bits 0-5 make sense. Sometimes, I see bit 6 set. What does this mean?
Reply With Quote
  #10   Spotlight this post!  
Unread 03-12-2016, 09:35 PM
fovea1959's Avatar
fovea1959 fovea1959 is offline
Herder of programmers
AKA: Doug Wegscheid
FRC #3620 (The Average Joes)
Team Role: Mentor
 
Join Date: Jan 2011
Rookie Year: 2011
Location: St Joseph
Posts: 385
fovea1959 will become famous soon enough
Re: Alternate viewer for Driver Station Logs (dslog)?

reawakening this thread. Was CSA this weekend, had a customer pop a main breaker on the field. Want to export the data so we can get average current draws out of the file (hard to estimate average current draw in spiky data).

Any documentation or reverse engineering on the current (<groan>) dslog format?
Reply With Quote
  #11   Spotlight this post!  
Unread 03-13-2016, 08:15 PM
Greg McKaskle Greg McKaskle is offline
Registered User
FRC #2468 (Team NI & Appreciate)
 
Join Date: Apr 2008
Rookie Year: 2008
Location: Austin, TX
Posts: 4,861
Greg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond repute
Re: Alternate viewer for Driver Station Logs (dslog)?

The data you are looking for is in the binary .dslog file. It starts with an I32 version. The current version is 3. Next is a LV timestamp which is sixteen bytes and described on our website if you wanna decode it. The rest of the files is a stream of entries every 20ms -- described below.

The numbers are often packed, and described as I.D(int.dec). For example, 7.1 is 7 bits above decimal and 1 below.

Trip time is a 7.1 int of milliseconds for round trip.
Lost packets is a byte of packets per half second, ranging from 0 to 25. So multiplying by four gives percent loss for the period.
Voltage is 8.8 volts.
CPU is 7.1 as a percentage.


Flags is a byte, with bits for
Robot Disable, Auto, Tele (what the robot reported running)
DS Disable, Auto, Tele (what the DS asked it to do)
Watchdog
Brownout


CAN utilization is 7.1 as a percentage.
Signal Strength is 7.1 as dB (not reported yet)
Bandwidth is 8.8 in megaBits (not logged yet)

PDP data is 24 bytes defined by the PDP.
First byte is an ID of which PDP is being reported.
Then 16 10 bit numbers (7.3) at bit offsets (8, 18, 28, 38, 52, 62, 72, 82, 92, 102, 116, 126, 136, 146, 156, and 166).
Remaining three bytes are resistance, voltage, and temperature with special values for scaling.

Hope that helps.
Greg McKaskle
Reply With Quote
  #12   Spotlight this post!  
Unread 03-25-2016, 03:11 PM
Aero Aero is offline
consumes pizza, produces code
AKA: Ari Lotter
FRC #0865 (Warp7)
Team Role: Programmer
 
Join Date: May 2013
Rookie Year: 2013
Location: Toronto
Posts: 30
Aero is on a distinguished road
Re: Alternate viewer for Driver Station Logs (dslog)?

Quote:
Originally Posted by Greg McKaskle View Post
...Hope that helps.
I'm having a hard time extracting the PDP data from the file.

As far as I can tell, after the initial 20 bytes of version and timestamp, each packet is 35 bytes long.
The PDP data starts at byte 11, and extends to byte 35.
Trying to decode this data with the offsets 8, 18, 28, 38, 52, 62, 72, 82, 92, 102, 116, 126, 136, 146, 156, 166 gives me values I assume are garbage. Trying to orchestrate a dslog file to give me expected values in the log viewer doesn't seem to work either. Is there a chance you could share the .vi of the decoding part of the log viewer?
Thanks!
__________________
2014 Waterloo Regional Winner (254 & 2056)
2014 Curie Semifinalist (118, 359 & 4334)
2014 Fall Fiesta Finalist (2056, 4476)
6 X Gracious Professionalism Award Winner
Reply With Quote
  #13   Spotlight this post!  
Unread 03-27-2016, 12:17 PM
Greg McKaskle Greg McKaskle is offline
Registered User
FRC #2468 (Team NI & Appreciate)
 
Join Date: Apr 2008
Rookie Year: 2008
Location: Austin, TX
Posts: 4,861
Greg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond repute
Re: Alternate viewer for Driver Station Logs (dslog)?

Reading your interpretation and rereading my description, I think I should clarify that there are ten bytes of nonpdp data, then 25 of pdp data.

Pdp has a one byte ID that would be useful in a multipdp robot, and then 24 bytes of measurements where the offsets are used for unpacking. If you were to apply the offsets without skipping the ID, that would indeed throw everything off. I pm'd about the VI

And one more thing, the sixteen channels are ordered backwards -- PDP[15] through PDP[0].

Greg McKaskle

Last edited by Greg McKaskle : 03-27-2016 at 04:15 PM. Reason: One more thing...
Reply With Quote
  #14   Spotlight this post!  
Unread 03-04-2018, 11:29 PM
tjf tjf is offline
Registered User
AKA: Tim Flynn
FRC #1257 (Parallel Universe), #1228 (RoboTribe)
Team Role: Mentor
 
Join Date: Jun 2016
Rookie Year: 2016
Location: Union County
Posts: 403
tjf has a reputation beyond reputetjf has a reputation beyond reputetjf has a reputation beyond reputetjf has a reputation beyond reputetjf has a reputation beyond reputetjf has a reputation beyond reputetjf has a reputation beyond reputetjf has a reputation beyond reputetjf has a reputation beyond reputetjf has a reputation beyond reputetjf has a reputation beyond repute
Re: Alternate viewer for Driver Station Logs (dslog)?

I was reading old threads and found this program: https://github.com/orangelight/DSLOG-Reader/releases

Some screenshots:

https://imgur.com/a/9Uy9B

Hope this helps!
__________________

1257 (2016) - Student
1257 (2017-Present) - Overactive Alumnus | 1228 (2017-Present) - I do stuff
KD2KRT
Reply With Quote
  #15   Spotlight this post!  
Unread 10-10-2018, 09:01 PM
prensing prensing is offline
Registered User
AKA: Paul Rensing
FRC #2877 (LigerBots)
Team Role: Mentor
 
Join Date: Mar 2015
Rookie Year: 2014
Location: Boston MA
Posts: 34
prensing is an unknown quantity at this point
Re: Alternate viewer for Driver Station Logs (dslog)?

I have written a simple Python command line script to convert DSLog files to a regular CSV file. This lets you convert multiple files without the complication of a GUI.

Note that I found and submitted 2 bugs to the DSLog-Reader projects which affect 2 of the status flags and the packet_loss value. With those fixed, the two routines produce the same value.

You can find the code in our Github repo: https://github.com/ligerbots/dslog2csv. The script works in either Python 2 or 3, but you need the "bitstring" package installed.
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 08:54 AM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi