Profiling a Robot

[MatthewC529]

Is anyone familiar with the best ways to profile the robot program remotely? I can access the robot over FTP and SSH but I am not sure how to go about profiling the program properly.

I am familiar with jstack so I was currently thinking of using that. Any other ideas?

I ask because I seem to have a memory leak or something similar that is causing the robot program to crash after some amount of time, being unable to allocate memory to continue running. This is not ideal. I work with a fairly large library written this year.

[GeeTwo]

Wow - running out of resources in just a few minutes; usually not a problem. If you don't find a proper resource monitoring tool, I'd suggest adding logging (printing to dashboard or equivalent) of every constructor - because that's where the resources are taken. I'm already concerned about how many resources many recent java programs seem to take and discard, when they could very easily reuse them. It sounds like you have such a condition, coupled with remnant references to the no-longer-needed objects declared earlier.

[MatthewC529]

Originally Posted by GeeTwo

Wow - running out of resources in just a few minutes; usually not a problem. If you don't find a proper resource monitoring tool, I'd suggest adding logging (printing to dashboard or equivalent) of every constructor - because that's where the resources are taken. I'm already concerned about how many resources many recent java programs seem to take and discard, when they could very easily reuse them. It sounds like you have such a condition, coupled with remnant references to the no-longer-needed objects declared earlier.

I am very aware, the library is large mostly because they are based on sound principles of pre-existing libraries such as Google Guava, LibGDX's Primitive Collections, JavaFX Property/Binding Framework and so on. As well as, of course, Robotics specific features.

The memory we use is approximately 20 MB at a time. We tracked the performance down to nothing in the library itself (which consists of 100+ files programmed by one person, too much to search without a profiling tool), but rather constantly updating and retrieving NetworkTable values as part of the IterativeRobot Loop.

I am not sure why, but I am guessing that is not the advised way of putting and getting values from a Network Table. Likely we will have to do it on an as needed base, a bit more complicated but not much.

Any suggestions for how to properly send and receive Network Table data?

[GeeTwo]

Originally Posted by MatthewC529

Any suggestions for how to properly send and receive Network Table data?

Seriously, this has never been an issue for us, and I can't think of a realistic reason why you would need to. What led you to the understanding that you manually needed to pass Network Table data between the 'RIO and the driver station?

[MatthewC529]

Originally Posted by GeeTwo

Seriously, this has never been an issue for us, and I can't think of a realistic reason why you would need to. What led you to the understanding that you manually needed to pass Network Table data between the 'RIO and the driver station?

I think you misunderstood what I meant.

In each update loop two methods were being called:

updateImmutableDashboardValues();
updateValuesFromDashboard();

The intent is for values that are not explicitly allowed to change from the dashboard are updated in the first method. In the second method we get any changed values from the Dashboard.

We are not directly interacting with NetworkTables in any way other than through the SmartDashboard and LiveWindow classes.

We will be doing a bit of testing, we think it is the way we used it that made it a problem. Tests will be done, likely at competition since they are quick and well.. we don't have access to the robot anymore.

[wireties]

Top is on the robot but thread-specific data seems not to work.

I am interested in profiling our solution also - some things seem to take longer that they should. Our code is a custom C++ setup using tasks for most everything and passing messages around (using pipes). I'll look into gprof and other tools, maybe try to get valgrind running etc. If I have any luck I will post.

[wireties]

Originally Posted by GeeTwo

Wow - running out of resources in just a few minutes; usually not a problem.

The OP is talking about the relative amount of time and real-time-ish resources, not the time of day or the passage of time on a clock. It can happen instantly and has little to do with the match length. The application code on the robot and the problem/solution domain is nothing like desktop or enterprise applications.

[fovea1959]

can you run visualvm on your driver's station and look at the JVM on the roboRIO? It has some pretty good tools for dealing with memory issues.

It's been a while since I've used it over the net, I don't remember if you need to do anything special when starting the JVM to be profiled.

[nickmcski]

Originally Posted by fovea1959

can you run visualvm on your driver's station and look at the JVM on the roboRIO? It has some pretty good tools for dealing with memory issues.

It's been a while since I've used it over the net, I don't remember if you need to do anything special when starting the JVM to be profiled.

If you do a java debug run it will run this command

env LD_PRELOAD=/lib/libstdc++.so.6.0.20 /usr/local/frc/bin/netconsole-host /usr/local/frc/JRE/bin/java -XX:+UsePerfData -agentlib:jdwp=transport=dt_socket,address=8348,server=y,suspend=y -jar /home/lvuser/FRCUserProgram.jar

That command should allow visualvm to connect to the roborio over port 8348

[MrRoboSteve]

First thing to look at is whether you're leaking heap. Enabling -XX:+HeapDumpOnOutOfMemoryError and then using jhat to analyze the result after you run out of memory may provide some information.

Note that I haven't run that procedure on Java on the roboRIO.

[fovea1959]

well, we just took a weather-related cancellation for tonight (AGAIN), so hopefully I'll be able to grab our test board and run some tests tonight.

@nickmcski, MrRoboSteve: is the "command to be run when debugging" defined in the build.xml in Eclipse, or is there a script on the roborio we have to fiddle if want to change the startup options when debugging?

[nickmcski]

Originally Posted by fovea1959

well, we just took a weather-related cancellation for tonight (AGAIN), so hopefully I'll be able to grab our test board and run some tests tonight.

@nickmcski, MrRoboSteve: is the "command to be run when debugging" defined in the build.xml in Eclipse, or is there a script on the roborio we have to fiddle if want to change the startup options when debugging?

I found the command in the wpilib folder on my computer but I know there is the same file on the robot, I have no idea which one it is using to launch the code though. Next time I can get my hands on our test electronic board I will fiddle around with it and see which file it uses.