Lvrt process causing CPU spikes

Title says most of it… we’ve lately seen the lvrt process eating about 30-40% of our CPU and it causes quite a bit of latency and lag on the robot, resulting in control issues.

Not sure what is causing it but I’m curious if others have seen this and what might be causing it.

1 Like

I haven’t seen CPU spikes, but I’ve seen LVRT use 10% of the total RAM. It’s kinda sad considering its only job in C++ or Java is to call frcRunRobot.sh in a subshell. Workarounds exist, like replacing lvrt with a custom script.

#!/bin/sh
APP_PATH=`nirtcfg --file /etc/natinst/share/lvrt.conf --get section=LVRT,token=RTTarget.ApplicationPath`
if [ "$APP_PATH" != /home/lvuser/natinst/bin/TBLStartupApp.rtexe ]
then
  exec -a lvrt ./ni-lvrt
fi
APP_BOOT=`nirtcfg --file /etc/natinst/share/lvrt.conf --get section=LVRT,token=RTTarget.LaunchAppAtBoot,value=true | tr "[:upper:]" "[:lower:]"`
APP_DISABLED=`nirtcfg --get section=SYSTEMSETTINGS,token=NoApp.enabled,value=false | tr "[:upper:]" "[:lower:]"`
if [ "$APP_BOOT" = true ] && [ "$APP_DISABLED" = false ]
then
  . /usr/local/frc/bin/frcRunRobot.sh 2>&1 >/dev/null
fi
sleep 1
3 Likes

Yeah - it was quite odd.

Robot code wasn’t restarting and it remained up and running but that process just kept spiking the CPU.

We’ve disabled things now but would love to know what it was trying to do and what we did to trigger it. It normally just gets ignored and sits idle.

1 Like

Is that what these spikes are?

1 Like

Maybe it’s trying to continually respawn the robot program. Since the second robot program will fail to initialize the HAL and exit, LVRT will retry in a tight-loop forever.

Ours looks similar but I’d have to grab the logs to be certain.

IIRC the reported rio CPU usage is the percent relative to the usr CPU %. Is lvrt a user task?

2 Likes

I’m honestly not sure what that is reporting but we’ve seen the process doing this with the top command. We thought at first it was somehow related to match data but we were able reproduce it in the lab now.

We think we are having the same problem, even with a blank code file. Did you find a solution?
Edit: We unplugged our CANivore which has all drivetrain motors and cancoders, and now the problem has gone away (still with a blank file). Problem is we need the CANivore because our utilization is too high otherwise.
Edit2: yeah so it’s not the CANivore. We basically found that our code causes the problem, and it continues to happen if you deploy blank code unless you power cycle. We assume some loop is continuing after our code ends but we can’t find it.

1 Like

Yes and no. We did two things.

We checked the box for “Disable RT startup app” in the RIO imaging tool.

We also removed the lvrt startup symlink.

Those two things resolved this for us and it hasn’t given us trouble since we hit the problem.

I would not advise most teams do any of this but it really depends on the code the team is running. This works for us.

So what does disabling RT Startup do? And by disabling the lvrt symlink I understand you followed the workaround from @calcmogul?

It disables the RT process from starting automatically:
https://knowledge.ni.com/KnowledgeArticleDetails?id=kA03q000000YGkDCAW&l=en-US

As for what that does beyond what it says - it disables something that is automatically running from NI and I’m sure someone who works for them can tell you exactly what but I’m not sure. Our thought was that it might prevent the lvrt process from running when we tried it. It didn’t seem to completely do that though so we also removed the symlink for it.

As for removing the symlink… no, not those instructions though I’m sure that works too. We deleted the symlink from the /etc/init.d/ directory. Again, this worked for us and I don’t know if it will for most teams.

If I’m understanding correctly:

  • Reflash the rio, disabling RT startup
  • SSH and delete /etc/init.d/lvrt
  • replace /etc/init.d/lvrt with @calcmogul’s custom script

Is this the correct process? I’m not sure if the last step is right

No. Definitely did not do this.

For us, the steps were to run the tool and check the box and hit the go button.

Then, once we rebooted, we observed the process was still doing something so we went a step further and deleted the symlink after noting where it was on the filesystem.

I meant re-image. But do you put the custom script in the same directory as lvrt or a different place?

We did not use a custom script so I can’t tell you what to do with it.

Oh, I see. Thanks!

1 Like

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.