CAN only architecture

[nuttle]

This is a great way to open things up, in the open source sense of the word.

In fact, the next time the control system is updated, I think a very good architecture would be to have a custom and closed radio for FRC, but to open everything else up 100%. Actuators would be the exception, but even these might be opened up at some point in the future.

The radio would be hardened for FRC, would have several 1G Ethernet ports (industrial connectors), and would also have several “safety” CAN ports (more below). The radio would support quality of service and enforce a per-priority-level bandwidth limit on the wireless transmit side (the field would do the same on it’s wireless transmit side). Because it is expensive to get FCC certification, the radio would be built around an off-the-shelf WiFi module and would likely have firmware based on OpenWrt.

The radio could also have antennas selected for optimal performance in the context of FIRST and would continue to handle things such as encryption. In addition, the radio would support connectivity between Ethernet and CAN, with quality-of-service and bandwidth management. Because of where it sits, and because it is closed, it could ensure no spoofing of safety packets on any of it’s CAN busses.

There would be a “standard” compute module which most teams would very likely use, but the safety case would not rely on any level of trust outside of the radio. The driver station S/W would have support for collecting diagnostic information from the radio, and also probably from the standard compute. For example, this module could be a future NI RIO. Significantly, something like CANifier could handle all of the I/O for things such as sensors, replacing the need for a custom FRC compute module with dedicated I/O pins.

All actuators would be required to be controlled via CAN, and to automatically disable within a short amount of time if they are not continually receiving enable packets on the CAN bus – there might not even have to be an explicit disable packet, but one could be defined.

At this point, the safety rule becomes that all actuators have to have their CAN port connected to a CAN bus with only approved CAN devices and a CAN port on the radio. Note that multiple CAN busses are good for reliability, especially in the face of wiring problems, and also for increasing the overall CAN bus bandwidth.

Any compute with Ethernet connectivity is allowed (weight, power, etc. rules still apply). It gets an Ethernet network on the robot and with connectivity to the driver station, and can also get to one or more CAN ports to communicate with actuators. Now, any compute is legal and safe.

At any rate, things done by teams along these lines will provide more of the good things that are part of the FIRST experience. I’m hoping things will explicitly move more and more in this direction. I’ve tried to send these suggestions in to FIRST previously, BTW. But, I’ve not been in a position to respond to any control system RFC, when these have been issued. However, I’m very happy to explain how this would all work, for anyone who is involved in things on this side of FIRST.

[Peter_Johnson]

@nuttle

All actuators would be required to be controlled via CAN

Some folks in FRC (eg team 971) care deeply about latency and hard real time behavior to create incredibly responsive control loops, and restricting actuators to CAN only would likely destroy that completely or relegate it to only the features that closed source black boxes provide (eg it would create a performance ceiling that is not currently present). The big advantage of the Rio is it provides the ability to do those loops in team software while simultaneously providing safety through hardware mechanisms.

Maybe if the main safety controller also provided time distribution and you could time tag commands to motor controllers? That would at least partially address the timing jitter problem, but I would think overall system latency would be quite a bit worse than is achievable today with Rio inputs and PWM outputs.

[nuttle]

Fair point – but the idea here is that the controllers would go open source as well. It would have to be covered in the interim though, or else this would have to roll out in tandem. However, one can do pretty well in terms of latency with good bus bandwidth/speed, particuarly with QoS.

[Peter_Johnson]

If the motor controllers are open as well, how is safety controlled within the motor controllers? It seems like doing that would just move the problem from the Rio to every motor controller.

I also see a big support challenge. Right now, CSAs at events mainly just need to know how to program the Rio and image other devices. Every unique device that can be customized with software creates a support burden that we don’t have today. Also, every unique device is going to have unique failure points.

[CarlosGJ]

@nuttle

the idea here is that the controllers would go open source as well.

I really doubt that’s going to happen. Allowing teams to write their own code on motor controllers would completely negate all of the failsafe enable/disable control baked into the RIO.

Edit: sniped

[nuttle]

Not really – it’s common to have heartbeat safety mechanisms where loss of a periodic signal causes a processor reset. This exact paradigm is widely used in industry.

Here are some references for anyone who wants a deeper dive:

Watchdog timer

https://www.edn.com/5-tips-for-designing-a-smart-watchdog/
https://www.digikey.com/en/articles/improving-iot-system-robustness-using-watchdog-timers

In short, one can get dedicated parts that do this processing and they add little cost or complexity these days. Things around this part are safety critical and locked (usually, in H/W) and this means S/W running on the processor is no longer safety-critical. I am confident this can be done, I imagine this would take the form of one or two open FRC controllers being released initially. It would depend on vendors choosing to follow this though. There are always trade-offs.

[CarlosGJ]

@nuttle

In short, one can get dedicated parts that do this processing and they add little cost or complexity these days. Things around this part are safety critical and locked (usually, in H/W) and this means S/W running on the processor is no longer safety-critical.

You’re talking about having both user-modifiable and locked code running on the motor controller, both of which control the H-bridge and are on the CANbus. This is basically a mini version of the RIO, which has the user-modifiable OS and safety-critical stuff on the FPGA. That’s a massive hardware and software scope increase compared to current motor controllers, which run on pretty simple microcontrollers (I think the Talon SRXs use dsPICs).

[nuttle]

I replied to another comment on safety – in short, it’s a solved problem.

The support thing is valid, though open source does pretty well in this area. I think the first thing would be to have a “standard” compute that’s similar to what is there today, and could therefore be supported much as things are now. Teams do use coprocessors now, and support here isn’t guaranteed. I suppose the way it would work would be teams would have to be sternly advised to consider this before choosing this route. To lessen risk, this would most likely have to be rolled out gradually, over a few years.

[nuttle]

Maybe. There’s a whole ecosystem of “safety” microcontrollers and techniques in this area. I don’t think they are radically more complex, in fact one really wants these to be isolated and orthogonal, so the main code is unchanged – this is what is being open sourced after all. One idea would be to have a locked part handling safety, and only allow F/W updates on the open processor. The safety part would be as simple and cheap as possible. Parts are quite cheap these days, and there are several candidates. One path could be to follow the CANopen safety work, there are parts that implement this function for CAN.

I can’t really speak to the scope increase, because I tend to work with more complicated systems, but my sense of things is this is quite manageable. The point is, there are viable alternatives. This would only come up for controllers that wanted to follow the open source paradigm, but I think such a controller could carry a marginal price premium.

[Joe_Ross]

Split from Is it possible to control CAN motor controllers from a coprocessor?

[EveryLittleBit]

With the advent of brushless control, the thought of teams writing controller firmware is making me preemptively smell burnt windings…

[nuttle]

I agree this could happen. It’s pretty intimidating if you look at just the config parameters on a Falcon 500.

However, it’s not as if no motors get smoked today, and most teams would likely either run not-open controllers or use validated firmware on any open ones.

There are H-bridge driver chips that have enable/disable which would be hooked to the safety processor ($2 gets you a 16-bit microcontroller with CAN these days), and they have shoot-through protection and current monitoring, so you could make the H-bridge as bulletproof as today (would likely not do reverse-polarity protection of power, for efficiency).

I think an interesting way to package something like this would be to have a single controller with 6 H-bridge legs, so either two brushless motors, three brushed motors, or one of each. Throw in a couple of pneumatic solenoid ports, and now one can control many interesting modules. This would help with number of PDP connections and also robot wiring. Total current would still have to fit through a single 40A breaker on the PDP. One wire pair for power, one for CAN, and possibly air, and you can run a swerve or many interesting mechanisms…

At that point, using a $30 Raspberry Pi for the controller might make sense. I’m not trying to design something here of course, just to point out that it’s a viable (and safe) option. The actual processor could even be opened up, if the controller had a few digital connections. I’m not saying this is the way to go, just a possible option.

Downsides pointed out thus far are cost (not bad at all compared to the cost of many controllers) and increased exposure to mistakes burning up motors.

Benefits include another opportunity for teams with the resources to go deeper, plus the usual benefits of open source – collaboration, the ability to fix bugs instead of being blocked by them, more people looking at the code, and very likely more features.

One could build robots pretty much as they are today, using not-open controllers and a RIO. Or, one could go open and distributed, with whatever compute and a couple variations of open controllers.