Alternate viewer for Driver Station Logs (dslog)?

[iambujo]

Hello - we are analyzing our ds logs due to lag issues seen at our first competition. I am wondering if the log structure/content is documented any where or if a portable simplified (windows or Linux) viewer has been identified. Ideally I'd like to view it in a way that I can create my own plots, crunch filter, etc without needed all the NI software installed. If methods exist already that would help!

Thanks

[Greg McKaskle]

The DS log file is binary (for compactness and has a simple header of a 32 bit version number that is always zero at this point and a (64,64) bit timestamp with an epoch of 12:00:00.000 AM 1/1/1904 (Universal Time). It is a 64 bit int of seconds since ... and then 64 bits of fractional seconds.

The rest of the file is a sequence of tuples (single, single, single, u8, i8) and I believe they are packed with no alignment. The attached code build the plots and the legend shows the plots being built.

The only complicated element that the last six plots packed into the u8. Each bit selects between Nan (which lifts the graph pen), and a value plotted on the scale. It is basically a bitfield of those six bits representing what the DS told the robot to run and what the robot ran during that time slice.

I think the last element to add is that the byte ordering is big endian.

Let me know if I left anything out or something doesn't seem right.

Greg McKaskle

[iambujo]

Wow, thanks for the reply Greg! I am not familiar with Labview so the image is a little 'greek' to me, but I am sure I can decipher it. If I have some free time (most likely post competition season at this point) I may write a little decoder app to expand the data and make it more portable for things like xplot and excel. Not sure if it will be useful, but I like a good challenge.

[fovea1959]

Greg: thanks for the information; I did a hex dump of some files tonight, your explanation makes sense...

it appears that after the initial time stamp, there is no timing information, and it's assumed that there is an assumed .020s between samples, regardless of reality? or are the samples written to the file on a .020s timer that doesn't have cumulative drift?

[Greg McKaskle]

The samples are written in a timed loop that will attempt to keep on the 20ms schedule, no drift. It runs at a high priority and doesn't have much jitter. By the way, we are now at version 1. I believe I posted details earlier this week, but the only difference is that the mode bits shifted down.

Greg McKaskle

[Joe Ross]

Originally Posted by Greg McKaskle

The samples are written in a timed loop that will attempt to keep on the 20ms schedule, no drift. It runs at a high priority and doesn't have much jitter.

Is this the same timer that is sending data to the robot? Have you characterized the jitter as the DS CPU gets overloaded?

[Joe Ross]

Originally Posted by Greg McKaskle

By the way, we are now at version 1. I believe I posted details earlier this week, but the only difference is that the mode bits shifted down.

I couldn't find that description of the new format. Would you mind reposting it? Also, what is the format for the dsevents file?

[Greg McKaskle]

The log file is not written out in the high priority timed loop. I try to keep that loop lean. It is done where we receive the status packets. What is logged is mostly the status info anyway. I have characterized the control packet loop with high CPU, but not the status or logging loop. The statistics on packet loss and latency are going to lag a bit just because of how they are measured and reported. The events tend to be even more lagged because they are often timeouts. But I have never seen a file where I thought the CPU usage messed up the data. If the DS doesn't have data for a time period, it replicates the new packet to fill in the void. So I would suspect that high packet loss would have flat spots for everything else and high bars for the packets.

The new format has version 1 and the trace bits are shifted down. Earlier in this thread, it shows a simple diagram that indexes the bits from the i8. It throws away the first two bits. We thought we were going to add more info and I modified code to prep. The result of that was that the bits are now packed starting at bit 0.

The dsevents file has the same header and then each event/warning is a timestamp and a LVString. The LVString is not NULL terminated, but is an i32 count of bytes followed by the bytes. The strings are possibly encoded a bit with <> tags, depending on who is sending the data to the DS.

Greg McKaskle

[Joe Ross]

Originally Posted by Greg McKaskle

The new format has version 1 and the trace bits are shifted down. Earlier in this thread, it shows a simple diagram that indexes the bits from the i8. It throws away the first two bits. We thought we were going to add more info and I modified code to prep. The result of that was that the bits are now packed starting at bit 0.

Looking at some data, bits 0-5 make sense. Sometimes, I see bit 6 set. What does this mean?

[fovea1959]

reawakening this thread. Was CSA this weekend, had a customer pop a main breaker on the field. Want to export the data so we can get average current draws out of the file (hard to estimate average current draw in spiky data).

Any documentation or reverse engineering on the current (<groan>) dslog format?

[Greg McKaskle]

The data you are looking for is in the binary .dslog file. It starts with an I32 version. The current version is 3. Next is a LV timestamp which is sixteen bytes and described on our website if you wanna decode it. The rest of the files is a stream of entries every 20ms -- described below.

The numbers are often packed, and described as I.D(int.dec). For example, 7.1 is 7 bits above decimal and 1 below.

Trip time is a 7.1 int of milliseconds for round trip.
Lost packets is a byte of packets per half second, ranging from 0 to 25. So multiplying by four gives percent loss for the period.
Voltage is 8.8 volts.
CPU is 7.1 as a percentage.


Flags is a byte, with bits for
Robot Disable, Auto, Tele (what the robot reported running)
DS Disable, Auto, Tele (what the DS asked it to do)
Watchdog
Brownout


CAN utilization is 7.1 as a percentage.
Signal Strength is 7.1 as dB (not reported yet)
Bandwidth is 8.8 in megaBits (not logged yet)

PDP data is 24 bytes defined by the PDP.
First byte is an ID of which PDP is being reported.
Then 16 10 bit numbers (7.3) at bit offsets (8, 18, 28, 38, 52, 62, 72, 82, 92, 102, 116, 126, 136, 146, 156, and 166).
Remaining three bytes are resistance, voltage, and temperature with special values for scaling.

Hope that helps.
Greg McKaskle

[Aero]

Originally Posted by Greg McKaskle

...Hope that helps.

I'm having a hard time extracting the PDP data from the file.

As far as I can tell, after the initial 20 bytes of version and timestamp, each packet is 35 bytes long.
The PDP data starts at byte 11, and extends to byte 35.
Trying to decode this data with the offsets 8, 18, 28, 38, 52, 62, 72, 82, 92, 102, 116, 126, 136, 146, 156, 166 gives me values I assume are garbage. Trying to orchestrate a dslog file to give me expected values in the log viewer doesn't seem to work either. Is there a chance you could share the .vi of the decoding part of the log viewer?
Thanks!