DataLogTool <3

I noticed the mention of DataLogTool in the latest WPIlib release notes. Couldn’t find any documentation but looking at the source it seemed interesting and easy to turn on, so we did.

Fast forward to our competition yesterday where nothing seemed to go right. In some post event analysis we wondered if our camera was getting confused by lights and reporting the wrong distances. We were able to go into the data log files and grep out the distance measurements, showing that it was behaving correctly.

Fine job, WPIlib team!

6 Likes

I’m glad you found it useful! Docs are here:

On my roadmap for next season is to build a full-featured offline data analysis tool (think Glass but for analyzing data files rather than real time display).

9 Likes

Does anyone have any recommendations on the best method or tool to create plots from the datalog data files?

1 Like

Today your best bet is likely Excel, or doing something custom with Python and matplotlib.

Plotly is a great option too – it lets you very easily create web-based graphs that you can zoom in, sync zoom and axes between graphs, etc.

2 Likes

Awesome tool, we have now used it to identify some instability in our shooter’s PID controller and fix it. Just a note for others who want to try this, it requires WPILib 2022.4.1 which was released in late Feb.

One question I have, what is the format for the “timestamp” when providing your own timestamp for logging? It is specified as a long, but no more information. I have tried using System.currentTimeMillis() which also gives a long (milliseconds since 1/1/1970) but this gives really strange times in the log.

It’s integer microseconds. On the Rio it grabs the FPGA timestamp (Timer.getFPGATimestamp) if you don’t specify the time. This is in the spec for the format but not in the docs yet; sorry about that!

1 Like

Thanks for that Peter, that makes sense now that you mention it.

FYI, I have been trying to encourage the kids towards a data driven analysis of robot performance issues and desperately needed this tool! We had previously been using System.out.println to output a comma separated list of values which we were copy/pasting into a spreadsheet and graphing. One of my biggest hurdles has been to convince them to take the time to extract and analyse data but they seem to always choose to spend hours “tweaking” things in the hope of resolving issues rather than spend 5 minutes extracting data which points them to exactly where to go. My aim is to have an easy to use trending tool that they can access quickly and easily, I am sure it will improve both productivity and robot performance!

I am currently writing a trending program that will open these log files and graph them, what is the scope for incorporating this sort of thing into WPILib itself? I note the general requirement for supporting both C++ and Java, but is this strictly applied to tools that run on computers and do not ever run on the RoboRIO itself, as they would still be accessible to every team? (my C++ is very limited but I have a heap of Java experience)

1 Like

Each of our tools is written only in one language. Most of our older tools are written in Java (Shuffleboard, SmartDashboard, PathWeaver, RobotBuilder) and most of our newer tools (Glass, SysId, OutlineViewer) are written in C++. This is partly driven by developer knowledge, and partly driven by application considerations (eg high performance real time graphing, access to C++ math libraries).

The project brainstorming board for my planned data analysis tool (“Sapphire”) is linked below. No development work has occurred yet. For performance and commonality reasons (shared widgets and graphing capabilities), and because I mostly work in C++, I’m currently planning on writing the tool in C++ (so it will look like Glass). But part of the point of a common data format is to enable development of tools, so if you want to develop something yourself, go for it!

1 Like

Why is this tool not pre built like all others? It took us a while to find the source and as far as we can tell no docs on how to build it. This was after we finally figured out that we had to update roborio and wpilib to have access to the API in the first place. A quick mention of needing the latest version and instructions on how to build would go a long way in reducing confusion. This really does look like a great tool but it needs to be accessible like everything else is with the Run / Start Tool in VS Code or better documented that it isn’t.

Did you install 2022.4.1 using the installer?

We did it through check for updates in vscode…based on your question I’m guessing that doesn’t update the tools?

You had to update WPILib since this feature was not added until a later release of WPILib. It was mentioned in the release notes for WPILib 2022.4.1 Release WPILib 2022.4.1 Release · wpilibsuite/allwpilib · GitHub

The release notes also explain how to properly upgrade WPILib.

That is correct. Checking for updates in vscode does not update tools. Those only get updated through the installer. I think that is noted in the docs for the check for online update vscode command.

We didn’t read the release notes. We started in the wpilib docs. The example couldn’t find the include so thought maybe we needed to update. Then we recalled seeing a “check for updates” in vscode. So we did that. We searched the file system for the tool. No luck. We searched github and found the source. And that is where we are.

Well that might explain it. Too many docs to read them all. Had a couple years off for COVID. I’m a bit rusty and apologize for my rant above given it really is that easy.

Good point, there does seem to be a few details missing about 1) WPILib update process/releases and 2) how this feature is new and only in v2022.4.1+. This could be a good opportunity to improve the docs

1 Like

Docs have now been updated to address both of these points, and a PR is open to add a bit more detail on the timestamp format and source as well. Apologies @kylelanman for the confusion this caused you; this is the first time we’ve added a new tool during the season and we missed the need to document the upgrade process more clearly for this case in particular.

3 Likes

I noticed that on one of our machines, but I was able to run it through ./gradlew DataLogTool (it showed up in ./gradlew tasks)

Just wanted to add, we made a quick and dirty plotting tool with Plotly. Add data log parsing · FRC2832/Robot2022-6861@9706e0d · GitHub

It is not the fastest (Python “arrays” are just lists), but should show how to plot Numbers (aka doubles) from the data files. It needs the datalog.py example file and our logic is in plotdata.py. It would be nice if DataLogReader also had a function for IsData(), instead of being the else case.

PS: This new DataLog feature has been GREAT, we have been able to easily look back at data and see what happened during a match (without having to steal the driver station and use Shuffleboard recording). We have been able to diagnose and fix our vision issues with it (since all NetworkTables are logged, and the Pi/Limelight uses NetworkTables :slight_smile: ) The hardest part is to figure out what to plot, as you don’t need all plots all the time.

One feature I would like added (maybe I need to add an Issue to allwpilib) is all the statistics that the FMS/DriverStation logs, can we add those to the logs also. Stuff like CPU usage, Loop Time, Free RAM, Packet Trip Time, Lost Packets, CAN stats, Wifi dB and Mbits, etc. This diagnostics are nice for us too :wink: