So my team has been using the Limelight this year for vision tracking and processing, but we have been encountering an issue with it seemingly randomly the entire season so far that we’re not sure how to fix it. Occasionally our Limelight will become completely unresponsive. No network table values are updated, the browser UI becomes completely inaccessible and our ShuffleBoard camera feed freezes and stops updating. We haven’t seen any errors in the console and the only way we know how to fix it is to power cycle the robot or the limelight. Clicking the Reboot RoboRIO button on the driver station doesn’t work because the Limelight is a co-processor. Has anyone else encountered this issue and solved it, or has an idea for how we could troubleshoot this?
We have another Limelight on another robot, and I believe I encountered this same issue on that one as well. Our original solution for this was to install a relay for the Limelight’s power that could be remotely triggered when it becomes unresponsive so we don’t have to reboot the robot, which would be especially crucial on the playing field. However, we soon realized the VEX relay we were trying first wasn’t going to work because it defaults to off when the robot is disabled as that disables roboRIO outputs, and have made a plan to move forward with a custom circuit that would default to on. We recently held a small competition and also encountered this issue twice on the field, and had to resort to lining up with a known distance from the hub each time to shoot for the entire match. Overall we’ve had many issues with this problem, and I originally thought that the issue probably stemmed from the fact that the Limelight was a Raspberry Pi at it’s core, and Raspberry Pi’s can have issues working properly and consistently sometimes maybe due to random memory errors or something. With our first regional coming up in just 3 days however, if there’s a solution out there somewhere that isn’t manually power cycling every time we notice it we would really appreciate it. Thanks
Be sure to set your limelight to use a static ip. If it is using a dynamic ip, that could manifest itself as you describe…with the limelight being unresponsive because it isn’t at the address that you think it is. This is a must.
Hm, I don’t beleive we’ve actually taken a close look at the LEDs when this happens yet, and was thinking when writing this post as well that we probably should have. I will make sure to take a look the next time this happens.
So the same issue happened again last night, I took a look at the status LEDs and they looked completely normal. The yellow stayed solid indicating static ip and the green blinked indicating it had a host.
I’d try a “ping” to the IP address, then maybe “ssh”. I can link instructions if that isn’t helpful… The idea is to try to work out at what point things are no longer normal (device is at the expected IP address, OS is running, etc.).
So we recently had our first regional competition, and although we did well, we encountered this same issue twice, essentially costing us a pre-qual match (we have shooting overrides for if this happens but would have the extra normal needed performance). After this happened at that on that important one, I made sure no-one power cycled the robot and checked everything that I could think of. The status LEDs were completely normal, yet the network table values, and camera stream didn’t update. The web UI once again became inaccessible when trying to access it from the static IP of 10.20.52.11:5801 and I ran the limelight finder tool and it found nothing. I forgot about trying to ssh to it’s ip address but I’m not sure I would have been able to if trying to access the web UI ip just said it was unavailable, and again the Limelight finder couldn’t find anything.
The next things I intend to try are to re-flash the 2022.2.2 image and ssh into it if this happens again, but because we can’t seem to find a reproducible cause of this issue I’m not sure how much more we can do to try and solve this ourselves. It seems like the bugfix of “Fix hang / loss of connection / loss of targeting related to open web interfaces, FMS, FMS-like setups, Multiple viewer devices etc.” was maybe intended to fix this, but if the update is working properly on ours it doesn’t seem like it was actually successful, and it’s still happening when it matters the most that it doesn’t.
My team had a similar issue where the Limelight would stop responding. We found that the Ethernet cord was coming a little loose. The Ethernet status lights next to the port would turn off, but jostling the cord would fix the issue. We hot glued the cord in as a temporary fix and are going to look into whether it’s an issue with the Limelight or our Ethernet cord. Don’t know if that lines up with your issue, but I hope it helps.
Just an update to our limelight…We had lots of problems with our limelight at KOKO-IN event Week 1. However…we did discover a couple of things…
We had a USB Camera attached to the USB camera port. Once we removed this, and moved the USB camera to the RoboRIO instead, the limelight started acting a whole lot better.
We was using Google Chrome to view the stream, however another thing that seemed to help was switching over to Microsoft EDGE. It seems Chrome times out the browser and you have to keep hitting “refresh” to get it to connect the stream to the camera again. Once we did these two things, the limelight did ALOT BETTER the rest of the weekend. BTY we are using image 2022.2.2 all weekend.
Hey guys, an update to this issue:
We believe we found the cause of the worst part of this issue, that being that we had the Limelight plugged into the 2nd port of our radio, which is apparently known to have issues pretty much randomly shutting down for some reason and applies to like the whole networking community rather that just FRC, which may be the reason they require the RoboRIO to be plugged into the first port. We made the change to plug it into out network switch instead, and it seems to have eliminated the issue of completely dying and not coming back on until we’ve power cycled the robot. However, now we’re still encountering the issue of losing connection to it, but more frequently and it reconnects after about 8 seconds. It’s less sever as the last stage but does appear to be happening a bit more and at any time. I found out I could pretty reliably check for when it happens by checking if the tl (latency being sent by the Limelight) has updated in a full second, and have watched it die while the robot was completely still both in teleop and while disabled. I’ve ran up to it a few times when this happens to check on the status lights and they still look completely fine, the same with the lights on the ethernet port. Each time after this happens we do get a message in console however, (I don’t have the exact message here but it reads something like) NT Server Connected: (limelight IP Address) port: (seemingly random port). Happy to know that it probably it shouldn’t make us lose matches anymore because of it but still an issue that we’d like to solve.
We do have our USB camera connected to it, we were having issues when plugging it into the roboRIO and displaying its feed at the same time as the Limelight on shuffleboard (neither camera would show up) and just chose that easy built in solution, so we’d hope not to remove it but will look into it, thanks!
You should remove the USB camera from your limelight. I can say definitively that streaming a usb camera through the limelight to the dashboard was causing the same crash issue you are seeing. Running it through the Rio or a pi with the wpilib image or photonvision (we found photon worked much better) fixed the issue for us.
Also consider if it’s necessary to stream the light light to the dash or not.
Have you tried running photonvision on the limelight? Under the hood the limelight is a raspberry pi, so it should behave the same and would let you only use one device instead of a second pi.