Team Fusion #364, Bayou Regional, FMS Woes

[otherguy]

2168 seemed to have this problem at NYC.

We never had an issue, period. Not for the entire competition until finals. We went 8-0-0, ranked #1, and had the highest teleop points. Our robot works, and we showed that. That's what makes this so unbelievably frustrating.

During our first semi-finals match... all three robots lost communication on the field. 2168 and 329 came back up but 1676 was disabled all match. The blue team did not see such an event. One of our mentors spent 10 minutes with the FMS manager diagnosing what was causing the problem. He said: "There was nothing they would do but it was very evident from the data the FMS collects that all robots were having 200 - 700ms delays."

1676 changed their bridge and Ethernet cable, and 329 deployed new code.
2168 verified wiring and swapped out one of our Jags which seemed to be failed on the drive train*** and the battery that we used during that match was getting way too low ~7V during the match. It was marked as good. (It is now marked as failed). We checked frantically for shorts but our chassis (as it had been all competition) showed infinite resistance to ground and pwr. The battery cable to the andersen connector was noticeably warm to the touch, andersen to the PD board was not however...

***Later we found, during pit testing, that the snap action breaker at that position seemed to be failing open. We have not yet tested the Jag we thought to be failed, it may have been fine, just not receiving power.

ultimately during our second semi-final match we had intermittent comms again. The lift motors were non-operational and since there was a loss of comm issues the drivetrain was not responsive.

Something to note, our team uses victors on the lift (pwm to digital sidecar on slot 2 of 4 slot crio). All other motors are CAN Jaguars connected to 4slot crio via DB9 connection. All 6 Jaguars remained operation during the matches. The victors were apparently not receving commands, however I was on the field right next to our robot and could see the light on the victors were solid, so PWM signals were apparently being sent?

All testing between matches on the side of the field showed all systems being fully functional. No faults with CAN traffic.

After jerking around in semi-2 we brought the robot back to the pit to try to figure out what could have been going wrong. Our operator noted that our button box (psoc3 FIRSTouch IO Module) was acting "weird". Typically in the IO tab on the driver station we should see all 16 Digital IOs pulled high. All indicators were extinguished, and not changing state with button presses, every now and then random ones would flicker on and off. Analog channels appeared to be responding properly to potentiometer changes.
We're using the classmate for the driverstation. It had not been restarted since the semi match ended.

Wiring for the board is all potted in glue, so its unlikely that it would have failed randomly. We checked wiring however and wiggling the usb connection at the psoc seemed to cause a fluttering of indications on the driver station. So we thought we had a wiring problem. I didn't believe we had a random failure of that board (The usb cable is knotted before leaving the box to prevent any damage inside to cabling), so I plugged it into my personal laptop and opened the driver station. Everything worked flawlessly. I plugged it back into the classmate and the same issue presented itself as before.
When the power cable was plugged into the classmate the indicators fluttering seemed to be more erradic which led me to think there was a noise issue on the classmate usb port or the hub we are using.
Restarting classmate (including battery removal) resolved the issue however, and it couldn't be replicated again.

We were able to replicate the field issue (the victors not responding - no ball lift) once in our pit, but over an hour of testing after that revealed everything to be operational.

We've come up with a list of items to check/replace at our next regional. PD board is at the top of my list, but that doesn't explain the weirdness we saw with the PSOC.

Has anyone had problems with the PSOC this year? Has anyone had success with other USB IO Modules? I know there was one which emulated a joystick (can't remember the name/seller) posted earlier in the season from a smaller robotics site.

Hopefully we can shed some light on what's going on here and come to a resolution. Good luck to everyone in fixing the issues on your bots. Keep us updated.