Grip running out of memory

[1024Programming]

Hi, we are trying to get Grip working for the 2016 game. We can currently deploy to the robot and view network tables because of the new update less than a day ago; however, when we deployed the code on the grip wiki to make tracking work, we got this error in grip when deploying:

Loading Dependency Injection Framework
Jan 23, 2016 6:54:11 PM java.util.logging.LogManager$RootLogger log
CONFIG: Configuration done.
Jan 23, 2016 6:54:12 PM java.util.logging.LogManager$RootLogger log
CONFIG: GRIP Version: 1.1.1
platform: /Linux/arm/
Java HotSpot(TM) Embedded Client VM warning: INFO: os::commit_memory(0xad400000, 131072, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 131072 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/admin/hs_err_pid10269.log

We interpret this error to mean that the roborio has run out of memory. If we load this code onto the robot and look at the driver station it says that it has 24M of RAM available.
PS: We have changed the code deploying address to admin instead of lvuser because of a radio problem earlier.

This is the code that we have downloaded to the robot:

package org.usfirst.frc.team1024.robot;
import edu.wpi.first.wpilibj.IterativeRobot;
import edu.wpi.first.wpilibj.command.Command;
import edu.wpi.first.wpilibj.command.Scheduler;
import edu.wpi.first.wpilibj.livewindow.LiveWindow;
import edu.wpi.first.wpilibj.networktables.NetworkTable;

import org.usfirst.frc.team1024.robot.commands.ExampleCommand;
import org.usfirst.frc.team1024.robot.subsystems.ExampleSubsystem;
import edu.wpi.first.wpilibj.smartdashboard.SendableChooser;
import edu.wpi.first.wpilibj.smartdashboard.SmartDashboard;
import java.io.IOException;

public class Robot extends IterativeRobot {
	private final static String[] GRIP_ARGS = new String[] {
        "/usr/local/frc/JRE/bin/java", "-jar",
        "/home/admin/grip.jar", "/home/lvuser/project.grip" };
    private final NetworkTable grip = NetworkTable.getTable("grip");
    public static final ExampleSubsystem exampleSubsystem = new ExampleSubsystem();
	public static OI oi;
	Command autonomousCommand;
    SendableChooser chooser;
    @Override
    public void robotInit() {
    	/* Run GRIP in a new process */
        try {
            Runtime.getRuntime().exec(GRIP_ARGS);
        } catch (IOException e) {
            e.printStackTrace();
        }
		oi = new OI();
        chooser = new SendableChooser();
        chooser.addDefault("Default Auto", new ExampleCommand());
//        chooser.addObject("My Auto", new MyAutoCommand());
        SmartDashboard.putData("Auto mode", chooser);
    }
    public void disabledInit(){
    }
    public void disabledPeriodic() {
		Scheduler.getInstance().run();
	}
    public void autonomousInit() {
        autonomousCommand = (Command) chooser.getSelected();
        
		/* String autoSelected = SmartDashboard.getString("Auto Selector", "Default");
		switch(autoSelected) {
		case "My Auto":
			autonomousCommand = new MyAutoCommand();
			break;
		case "Default Auto":
		default:
			autonomousCommand = new ExampleCommand();
			break;
		} */
    	
    	// schedule the autonomous command (example)
        if (autonomousCommand != null) autonomousCommand.start();
    }
    @Override
    public void autonomousPeriodic() {
        Scheduler.getInstance().run();
        /* Get published values from GRIP using NetworkTables */
        for (double area : grip.getNumberArray("targets/area", new double[0])) {
            System.out.println("Got contour with area=" + area);
        }
    }
    public void teleopInit() {
		// This makes sure that the autonomous stops running when
        // teleop starts running. If you want the autonomous to 
        // continue until interrupted by another command, remove
        // this line or comment it out.
        if (autonomousCommand != null) autonomousCommand.cancel();
    }
    public void teleopPeriodic() {
        Scheduler.getInstance().run();
    }
    public void testPeriodic() {
        LiveWindow.run();
    }
}

Sorry for long code. If you have any reason for why the robot is running out of memory, please share with us.

[SLAB-Mr.Thomas]

Me too! I was about to post this same error. We use C++ with the SampleRobot. The Java runtime for the grip gets the error when we execv it. Tested Java runtime with a Hello World and that worked fine.

Can't get grip to work on the driver station laptop either - don't know how to get a roboRIO USB camera image to it so no grip for us right now.

Now the next day and this is brutal but more hours of frustration and I out lasted it and got some results.

Doesn't seem to do the same thing 2 times in a row but the out of memory errors subsided and the robot program and the TableViewer are able to see changing values in realtime.

GRIP is running on the roboRIO using the USB camera.

What seems to give the most repeatable runs was in the driverstation GRIP pipeline development turn off all the eye preview images, then download the grip files, and close GRIP on the driverstation.

Then on the roboRIO deploy the robot code.

Still, rerunning GRIP sometimes gives varying results such as out of memory or Port 175 error 111 - connection refused. Rebooting everything helps.

GRIP leaves a lot of trash on the roboRIO in the /tmp directory and the /home/lvuser directory. Tedious going through tmp deleting files.

Other than some random crashes and failure to restart cleanly for unknown reasons, the only bug that's obvious is selecting ContoursReport publishing values area and height creates NetworkTables keys height and centerY usually and sometimes solidity and height.

I did have to change the sample program a bit:

auto grip = NetworkTable::GetTable("grip");

had to be changed. The "auto grip =...." was brought into the class, made a smart pointer and initialized in the constructor initialization list. Then it worked. Couldn't resist a few other "improvements" to the sample but they probably only help troubleshooting and didn't make the code run any better.

GRIP could be a valuable vision processing program when it's stable. Hope that is very soon so we can use it.

More thinking: Conjecture about one of the errors related to unable to connect. When the roboRIO starts its robot code the GRIP process is created, too. When another robot program is deployed in Eclipse it starts immediately but has a connection refused error. My conjecture is the GRIP process from the previous robot code start is still running and attempts to start another such process for GRIP fail. If I manually kill the grip process and deploy again, it works okay.

How can we get the GRIP cleaned up to allow another process to run?

And I'm still stewing about all the trash files left behind each GRIP run and am concerned about that GRIP log that is continuously being written to.

[1024Programming]

Thank you for your input on the problem. When I get to the shop today I will try to do the preview thing and deploy. When you talked about cleaning the /tmp folder on the roborio, how exactly do you do this? I agree that this will be very tedious, but we just want to be able to test until a new update comes out.

[SLAB-Mr.Thomas]

Attached file has "good" and "bad" runs as displayed in NetConsole.

Our robot.cpp and project.grip files that work (much of the time) are included as listings not as files attached.

I estimate that about 1 in 4 runs gets out of memory for Java for unknown reasons - seems random to me. Always fails after deploying from Eclipse then about 1 in 4 fail after that.

We use PuTTY to access the roboRIO's Linux console. Some displays of the files with error messages/dumps and processes are included in the attachment.

[1024Programming]

I know this is a little thing, but do you have a java version of the code? Is that the sample code from the grip wiki on github or did you modify it? If it is straight from github, then I know where to find the java version.

We use PuTTY to access the roboRIO's Linux console. Some displays of the files with error messages/dumps and processes are included in the attachment.

Yea, I was wondering if maybe putty would be the thing for that.

[SLAB-Mr.Thomas]

We don't use Java on our team. Had to install JRE on roboRIO just for GRIP - our first experience with Java there.

I copied the GRIP C++ example but one line had to be changed to prevent instant crash - the line declaring the NetworkTable variable "grip" has to be in the class. Otherwise, I think the example worked well enough.

The rest of my changes to the example were to help debug the various error messages that seem to come and go.

I wonder if the WPI GRIP team is monitoring these posts. I've mentioned 2 errors I haven't seen posted elsewhere. Those random occasional out of memory are a real pain and could be a deal breaker for us using GRIP and I really want to use it. Incorrect labeling of report variables needs attention, too.

Other errors I've gotten have been mentioned elsewhere, I think.

[ThomasClark]

Originally Posted by SLAB-Mr.Thomas

I wonder if the WPI GRIP team is monitoring these posts. I've mentioned 2 errors I haven't seen posted elsewhere. Those random occasional out of memory are a real pain and could be a deal breaker for us using GRIP and I really want to use it. Incorrect labeling of report variables needs attention, too.

Other errors I've gotten have been mentioned elsewhere, I think.

I really recommend running GRIP on a coprocessor if you're running out of memory a lot. The roboRIO only has 256 MB of RAM, and most of it's taken up the LabVIEW runtime and your robot program. The Kangaroo PC is a really good option, although people have also gotten GRIP to run on a Raspberry Pi.

Can you describe the incorrect labeling problem? I do monitor Chief Delphi for GRIP-related stuff, but you'll get a faster response if you open an issue.

[Ken P.]

Does the RoboRio system restart GRIP when the robot restart command is sent? Another mentor and I were wondering if the restart command would free the resources in use by GRIP or if you then have 2 sessions running at once.

[ThomasClark]

Originally Posted by Ken P.

Does the RoboRio system restart GRIP when the robot restart command is sent? Another mentor and I were wondering if the restart command would free the resources in use by GRIP or if you then have 2 sessions running at once.

Restarting the robot should kill all instances of GRIP. It'll start back up again if you deploy or if your robot program automatically starts GRIP.

Also, as a general response to this thread, I've found that some extra JVM options (-Xmx50m -XX:-OmitStackTraceInFastThrow -XX:+HeapDumpOnOutOfMemoryError) help with OOM errors. The next release (v1.2.0) will have these by default, and allowing custom JVM arguments when deploying.

[1024Programming]

Originally Posted by ThomasClark

Restarting the robot should kill all instances of GRIP. It'll start back up again if you deploy or if your robot program automatically starts GRIP.

Also, as a general response to this thread, I've found that some extra JVM options (-Xmx50m -XX:-OmitStackTraceInFastThrow -XX:+HeapDumpOnOutOfMemoryError) help with OOM errors. The next release (v1.2.0) will have these by default, and allowing custom JVM arguments when deploying.

How do I make the robot automatically start GRIP from the robot code?

[Greg McKaskle]

In one of these situations where you are running out of memory, can someone shell in and run top to see what memory is being used by various processes. I don't really expect that this is because of the LabVIEW runtime, but the numbers will tell the tale.

Greg McKaskle

[brk]

Originally Posted by ThomasClark

Restarting the robot should kill all instances of GRIP. It'll start back up again if you deploy or if your robot program automatically starts GRIP.

Also, as a general response to this thread, I've found that some extra JVM options (-Xmx50m -XX:-OmitStackTraceInFastThrow -XX:+HeapDumpOnOutOfMemoryError) help with OOM errors. The next release (v1.2.0) will have these by default, and allowing custom JVM arguments when deploying.

I'm running v1.2 already, so does it have these options already? I was going to change the launch code to try to tweak the memory, but if that's been done already in v1.2 it wouldn't be worth it. I see the GRIP jvm is consuming about 185-190M of the 256M of RAM. The Robot & Network programs are the next highest consumers totaling about 50-55M, so its definitely GRIP chewing up all the RAM. We've used the Beaglebone Black in the past for image processing, but really wanted to try keeping it simple and use the rio. Is there a chance v1.3 is coming out soon and addresses the memory issue?

[medofbr]

Originally Posted by ThomasClark

I really recommend running GRIP on a coprocessor if you're running out of memory a lot. The roboRIO only has 256 MB of RAM, and most of it's taken up the LabVIEW runtime and your robot program. The Kangaroo PC is a really good option, although people have also gotten GRIP to run on a Raspberry Pi.

Can you describe the incorrect labeling problem? I do monitor Chief Delphi for GRIP-related stuff, but you'll get a faster response if you open an issue.

We are running into the out of memory problems and are considering using the Kangaroo PC or Raspberry Pi. Can others comment on what levels of success they have had with either of these two approaches?

Some of the people were talking about this on Gitter.im. In order to prevent the entire JVM from crashing you may need to only allow "-Xmx50M" as the runtime arguments while deploying GRIP. GRIP does also prefer to have ~90MB of RAM if it's available. Otherwise, you can always work with what you have (adjusting the RAM by 1MB until it stops crashing). "-Xmx50M" -> "-Xmx55M" -> ... -> "-Xmx128M"

There is a confirmed system with a raspberry pi 2, but sometimes the system claims to be running at ~200% CPU usage. It's better off to ignore what the Pi thinks and simply test the functionality when it comes to the raspberry pi 2.

Good luck! Feel free to stop by the gitter.im chat if you have any more questions about GRIP.