Tonight while running our robot, our vision stopped working. After looking at the logs, I found that half of the config files were gone. It lost the configs for the pipelines and one camera.
Any thoughts on why this would happen? Nothing vision wise has been touched for several days.
Interesting. We haven’t gotten any data loss reports before. Is this on a Pi with SD card? If so, did you use the Pi image? If not, what platform, installation method, storage medium, and filesystem are you using?
The default Pi image uses a journaling filesystem which should stop most types of power-off data corruption (not loss), and we’re typically pretty careful in the code to flush open files when they’re changed. Because of how I understand the journal works, I don’t think it makes much sense for the files to get deleted on a journaling filesystem (only losing some changes makes sense), even if we have them open on power off. If you’re using an SD card and a journaling filesystem then it seems like either your SD card is going bad or you got really unlucky with the power off.
We are using a Raspberry Pi with the Photon Pi image on a SD card. The logs shows that it shutdown fine, but then on the next bootup it shows:
Could not load camera /opt/photonvision/photonvision_config/cameras/mmal_service_16.1’s config.json! Loading default
and
[2021-01-11 22:17:12] [Camera - VisionSourceManager] [DEBUG] Matched config for camera “USBFrameProvider - opencv_CargoCam” and loaded 0 pipelines
[2021-01-11 22:17:12] [Camera - VisionSourceManager] [DEBUG] Matched config for camera “AcceleratedPicamFrameProvider” and loaded 0 pipelines
Original It was loading 3 and 2 pipelines.
They were running the robot with low batters to the point at which the bot voltage would drop to ~7v while accelerating. That makes me uneasy with the Pi especially.
I’ll load another SD card and see if it happens again.
The Pi filesystem should be robust to sudden power-offs. But I think we’ve heard a few reports of this at this point. Do you know any specific way to reproduce it? I can try with my Pi at home.
I don’t know for sure. The night we lost ours, the robot was being driven on a pretty low battery such as that when they tried to move fast it would physically stutter from power drops. My assumption is that when this happened the pi had a brownout and then was trying to boot back up but kept getting shut off in rapid succession with each stutter of the robot.
I realize this won’t be an issue at competition since we shouldn’t be running low batteries. With more diligence in changing batteries while practicing it can also be avoided.
I don’t think it is necessarily a Photon issue, more just that we were putting it through an abusive situation.
Maybe this line here is doing something?
We delete the old camera config directory before writing all the new ones, to make sure that old camera config files don’t stick around. I guess it’s possible that if power was lost between when that happened and when the config was actually saved you could see this? You could check your logs for logs where you do see “Saving to disk…” but not “Settings saved!”, though who’s to say that your log file also wasn’t corrupted.
In any case that code path is only taken if you change settings, so if you were just driving that shouldn’t happen.