Background: In this thread about Jaguars vs Victors I suggested a design competition for FRC teams to design a new integrated control system to replace the current one, which many feel is too disjointed and complicated. That discussion started to dominate the thread and I decided to form one here to hold that discussion, the relevant part of my original post is below:
Personally, I’d really like to see FIRST open up the control system with a design competition. Let teams design and submit control system ideas, designed around what teams want/need and their resources and let the FIRST community and it’s thousands of brilliant minds weigh in. We already have a competition to design robots in the winter and spring, why not control system summer and fall?
While the guys at NI and TI have done an admirable job, I do agree the current system is an amalgam of several good ideas with bits hacked together to work in a manner that’s… well let’s go with frustrating. But those guys were small in numbers, and probably did not have all that much to work with. I sincerely think the mentors and students of FIRST stand a good chance of coming up with an effective, workable solution to our somewhat oddball use case. Even if only because of the sheer amounts of brainpower we have at our disposal.
So, what have you guys got? Maybe we should start with a list of basic requirements for the system, I’ve quoted part of Al’s suggested requirement from the original thread:
A good alarm on the robot that traps any voltage drops below a specific threshold and reports it when something bothers to check (I built one so I know this can be done and how handy it is). Mind you, not merely an analog to digital converter, a latched alarm that doesn’t depend on being checked.
The ability to separate the power for the control logic from the power for the motors (at least for troubleshooting, I can power all the Propellers on my test frame with a pair of D cell batteries if I wanted).
The ability to store data on the robot that will survive a complete power down (already built that with flash). Why? Cause when things go nuts you can read that memory and unlike with the WiFi you don’t have the overhead of reliable TCP/IP and buffering up.
The ability to check the quality of the link from the field control (that’s a ghost that’s caused enough headaches). Again, probably gonna need the memory for that…cause right now I can’t tell you after the robot powers down if you lost the field.
I2C, SPI or serial (emulated is fine, but with completely functional examples for the software).
6A. Circuit protection that is consistent. So a 40A breaker that actually trips at 40A and does so over and over…not once at 40A, once at 30A, once at 20A and then maybe will be like that for a while.
or
6B. A latching current monitoring system that can tell you if you trip the breaker from over current.
(I’ve already built 6B and frankly so can anyone…look at the TI appnote for the current monitor in the black Jaguar.)
This is pure programming and controller design:
It must be able to do more than one thing at a time…and not because your a wizard that can hand punch player piano roles (though if you can get your robot to run on player piano roles post it LOL). It is surprising difficult to thread with the cRIO as configured for FIRST…it does work…it took a while to figure it out.
Course the Parallax Propellers eliminate this concern a whole different way…you just keep adding chips and cogs until you have enough and they can all run at the same time (communication between them is a bit more tricky because of it).
The connectors have to latch. There’s nothing quite like a great robot failing because a poor connector slipped off.
Documentation. I know that the reason no one will find the blue prints for human kind’s greatest creations is because someone drew them on napkins during lunch and then left the used napkins in the garbage, but we need people to learn this stuff and guidance from the people with the most experience with it is always welcome. Leverage technology and use a Wiki if you must. Also, JavaDocs with notes like “does not work” is not what I mean.
It has to be cheap enough to ship to all the teams quickly and safely, and it has to be economical enough to send teams a baseline system within the existing funding constraints (the new cRIO has lowered the bar on the total package price by a fair bit, thanks NI).
>NO< undocumented connectors. You too can keep the smoke in my ears and in your electronics. Just make sure people understand what connects to what clearly. Oh and send at least test cables for something like the Jaguar with a bill of materials.
(Partially moved the contextually relevant parts from old topic…)
I think that video capability should be optional (that’s why it didn’t bother me when I built my Parallax Propeller based test platform I have that the Parallax Propeller’s USB host capability is sort of slow for webcam use…I figured a laptop would do it if it was needed…sort of like you add a compressor and tank for pneumatics if you want them). I think if you can remove features you don’t want…and are resource expensive like this…the more variance you can have in what you use that weight savings for elsewhere. Some teams might not want to tackle the challenge of video processing (even though libraries like PyGames are making that easy as Py :p)
So I would think flexibility…but not at the expense of reliability or practicality should be a priority of some sort.
I see no mention of a system that is quick to boot and establish communications. For whatever the reasons may be (there a few threads discussing it), the current 802.11 scheme takes a while and has plenty of issues besides. So if that means scrapping 802.11 for a proprietary radio link IFI-style or modifying the wifi setup to streamline things, whatever. The general requirement is quick boots and quick and rock solid connections.
Also it’d be nice if the controller booted faster…
I’m not sure, but I think there was a point in there about boot up time
I agree, Wi-Fi has too much overhead in general for this setup, including the time for the router and everything to start up, connect together… etc. I’d really actually like to see an 802.15.4 based system, but I’m not sure it has the bandwidth for the full video streaming everyone is spoiled by now (In my day we didn’t have streaming video, we drove our robots with our eyes! ;)) but something could almost certainly be worked out. Maybe there are Wi-Fi based solutions that are simply faster and more streamlined?
I do think the system shouldn’t take so long to start up, but I’m spoiled by the IFI system that just started running, so perhaps the boot up time isn’t as long as it feels to me.
I’ve transplanted my comments from the other thread here as well:
The old control system had a back up battery (7.2V) that was added specifically to prevent motors from browning out the controller and radio. The boost converter on the PD board we have now does a pretty good job of protecting the controller from blackouts.
If you meant just to have them connected to separate power so you can turn off one without the other, then just use a separate battery and PD board to run the controls and tie the negatives together.
You can do this by monitoring the wi-fi signal strength and ping latency, but those aren’t great indicators, but if we’re going to be stuck so many layers away from the actual radio hardware, we don’t have any accurate link integrity data.
The 2005 breaker board had a serial port on it that would tell the controller about breakers that popped, it was pretty useful. I’ve heard rumors the current PD board was supposed to have that, but it didn’t make the cut. I suspect the people making the decision underestimated just how useful that data is.
My test robot runs DD-WRT. Hack the routers, streamline the protocol or just run UDP if you don’t want the overhead of TCP/IP. So what if it ends up being 802.11ish…that’s just a few million sources of interference you don’t need to bother with. Besides there’s no rule you have to transmit IP at all, or even this sort of traffic at any level in that particular range. Course eventually you’ll get the FCC’s attention if you go too crazy, but that sort of stuff shouldn’t need to happen for what you ask.
Given the (very) real-time nature of what we’re dealing with, UDP is probably best. I mean really, if the robot loses a packet, by the time it figures it out, the data is ancient history. I’d retain TCP for important things like software downloads and such, where losing data is actually quite bad.
I suppose, Wi-Fi itself isn’t really the problem, but more the actual radio hardware setup we have now, so that could probably be improved on without being replaced entirely.
I’m suggesting this system wide basically. So that everything with control logic (including the Jaguars and Victors) can have that logic powered up from a separate source from the one that powers the motors.
You can do this by monitoring the wi-fi signal strength and ping latency, but those aren’t great indicators, but if we’re going to be stuck so many layers away from the actual radio hardware, we don’t have any accurate link integrity data.
Actually I’d just like packet statistics for the control system. Especially if there’s a laptop on the robot it’s quite doable. You don’t need to go quite to the point of a whole WireShark packet sniffer, but you could at least tag the types and quantities of packets sent and received that would be a great help.
Speaking of which I’d like a real time clock in the control system please, regardless of the presence of the laptop (epoch time is fine and it doesn’t need to be from 1970 like in Unix).
The 2005 breaker board had a serial port on it that would tell the controller about breakers that popped, it was pretty useful. I’ve heard rumors the current PD board was supposed to have that, but it didn’t make the cut. I suspect the people making the decision underestimated just how useful that data is.
Really would like to have confirmation of the magnitude of the current that did the tripping. I realize this adds weight, so as I originally planned and did these could be modular. I think they are most important on drive systems and other places where occasional overloads might happen and get misdiagnosed.
In point of fact someone will point out that a DMM can do this (but slower) or an oscilloscope with a current probe can do this, faster but much heavier. I’m looking for the compromise. Something light enough for competition, approved for competition, but removable for those that don’t want it. Like the suggestion I gave for reverse voltage protection on the Jaguars.
Aha, I see, it makes sense, but I’m not sure it is really weight feasible to add a secondary power source like that, and I’m not sure what advantage it would really get you?
I think the current FMS software tracks that data, you might be able to get your hands on it if you ask the right person, perhaps Mark McLeod? He seems to have a lot of background info on what’s going on there. An RTC is a nice feature, but for the 3 minute bursts of activity is it really necessary?
Since the over-current protection in Jaguars is software based, it would easy enough to save the value when the shutdown activates. For the breakers, it would have to be another solution obviously.
Testing mostly and perhaps the elimination of replicated parts (instead of 10 speed controls trying to regulate their control logic power, you have 1 cleanly regulated power source that does the whole step up and down once). If you put a COTS computing device on the robot now you add a second power source (the battery in it).
I think the current FMS software tracks that data, you might be able to get your hands on it if you ask the right person, perhaps Mark McLeod? He seems to have a lot of background info on what’s going on there.
Thanks, I’ll follow up on that after some sleep.
An RTC is a nice feature, but for the 3 minute bursts of activity is it really necessary?
For logging, especially if there’s a place to store the logs, it could help. I mean you can do it in software now…it’s not incredibly accurate but it works and it works well enough for competition purposes. I just figure if you have a decent time source it’s one less thing someone new had to figure out (besides my little flash logging module already has an RTC for this whole reason…I didn’t want to have to rely on more code in the system you were logging if I could avoid it).
Since the over-current protection in Jaguars is software based, it would easy enough to save the value when the shutdown activates. For the breakers, it would have to be another solution obviously.
Matt
Yeap, the Jaguars can collect the data, but where on the robot can you put it so that it survives once you turn off the power? You can send it back to the driver’s station if the communications is cooperating. Also this requires checking that current sensor which I’m sort of trying to avoid as it adds to the traffic on the CAN. Really it doesn’t need to be something you have to poll. It’s easy enough to make a circuit that simply latches if the current exceeds the threshold (so it can respond quickly without interrupting what you are doing…and much faster than the circuit protection). I’m proposing something not so clever as a digital to analog converter…more a circuit that produces a simple on or off with a reset (flip-flop). So off is under current limit, and on is exceeded the limit…feel free to reset and try again.
The nice part is, even a novice programmer could monitor that and know if they went over the current limits…and more importantly then they can decide what to do with that event before they end up tripping the circuit protection. However, and more importantly, it can be on the robot while it moves and it just immediately removes the doubt if you are over those current limits and how often it happens. It’s doesn’t even really matter if it’s a Jaguar. Heck hang an LED off it…you see the light start wondering why.
I see, but I’m still not convinced it would help enough to warrant the extra power source. And a COTS computing device doesn’t necessarily have a power source, especially if you use a small embedded mini-ITX board or something.
I am convinced, being able to log and time data offboard would be useful, actually there were a couple times we were debugging an intermittent task crash this season that would have been useful. Once the system dies, there was no way to get the information we needed. It was a lot of blind guessing, and a lot of ibuprofen.
I see, I guess I was thinking too high level (monitoring the current and logging spikes and dips and breaker pops), you just mean detecting and recording a transient over-current system. It might be worth considering, in addition to a digital signal indicating the spike, perhaps one that indicates voltage present at the node with a comparator (to determine if the breaker is currently open or closed, maybe a moot point with self resetting breakers?) and a reset line to clear the latched fault?
That’s very true, but it’s allowed if you want it (so far as I can tell at least).
The way I look at it, I can do this right now (and on two I did)…but I probably have to ruin the speed control for competition and I see no reason for that.
There’s no reason you couldn’t use the main battery for everything if you really want to and don’t mind the issues from the motor. In point of fact, if you separate the control supplies and the motor supplies you could just use a big capacitor on the existing battery. The control logic doesn’t require much (a couple of C or D cells is enough for what I have). A capacitor could probably provide just enough ‘ride-through’ for a short time. Actually, if you wanted you could get a little exotic and use a LiPo battery back for that if a suitable charger was available cheap. Then it would be cell phone weight.
Actually for fun my existing test frame has a Boss Audio 3.5 Farad (you didn’t read that wrong) capacitor on the frame with the voltage meter ‘cap’. The whole reason it was there was to try providing a lower internal resistance to hold up the battery if the speed controls get too much and cause a brown out. Just something I was playing with and it was less than $100. (Yes now besides stick welding with the robot’s battery you have a capacitor to insure you can lightly weld with your robot as well…lol)
I am convinced, being able to log and time data offboard would be useful, actually there were a couple times we were debugging an intermittent task crash this season that would have been useful. Once the system dies, there was no way to get the information we needed. It was a lot of blind guessing, and a lot of ibuprofen.
I see, I guess I was thinking too high level (monitoring the current and logging spikes and dips and breaker pops), you just mean detecting and recording a transient over-current system. It might be worth considering, in addition to a digital signal indicating the spike, perhaps one that indicates voltage present at the node with a comparator (to determine if the breaker is currently open or closed, maybe a moot point with self resetting breakers?) and a reset line to clear the latched fault?
Matt
Right now my logger prototype uses the CAN bus and has an RS232 level serial port as a back up (so you can put it on a cRIO serial port if you tinkered a bit, or put it on the CAN bus). I originally built it both to monitor the CAN bus and because I figured CAN was more direct than bit banging out serial on digital I/O.
Yes I agree, there are going to be times when under or over is not enough. Right now my current monitor uses a window comparator set with fixed resistors so you can actually tell if you have too much or too little (if you want…otherwise ignore one side). The same with my voltage monitor. I thought about using ‘digital potentiometers’ and I tried it…but I’m not sure the extra cost is all that necessary when you can get cheap good enough resistors at RadioShack. Besides the worst that happens is someone uses the wrong resistor and it trips at 60A instead of 30A…it’s not the circuit protection you’ll live. People just need to learn Ohm’s law or use a look up table and learn how to read color code (actually right now the resistors can be fixed into a connector so you don’t even need that, could just get the connector with resistors in it for that setting).
Course I used rail-rail JFET opamps so there’s the whole slew rate business, but still that’s much faster than a DMM and much more practical in this application than an oscilloscope. I actually previously discussed basically what I’m talking about here in these forums. I could have done this with discrete transistors but even with surface mount I’m not sure that saves me anything over the integrated package…besides then there’s that whole SWARF! thing and more possible points of contact.
I think in order for this to useful you need the reset (set reset flip-flop (or one made of gates) or JK flip-flop). Otherwise basically all that the thing is good for is the first over/under limit. Hopefully this is set below the circuit protection threshold and it actually doesn’t matter if it’s above the battery negative…it’s easy enough to clamp the high impedance input if it’s put in backwards. Though back to my previous point I’d prefer that putting things in backwards requires at least as much effort as a square peg in a round hole to achieve.
I’m doing my best to follow this thread, and I may ask clarifying questions or point things out from time to time.
Logging was mentioned. The DS now has a Charts tab that displays the packet loss and trip time for communications – same as the field has. It displays CPU usage, RAM, flash, and system voltage. It doesn’t have any current data available to log. The robot flash is capable of logging info, but I’m not sure how much data or what data should be logged by default.
The DS also logs the mode that the field or DS asked to be run and if the robot code is instrumented, it logs what code actually ran on the robot.
The DS data is saved on the DS and a log viewer is in the Program Files/FRC Driver Station folder.
What concerns me is that during several matches last year we had a situation where the robot clearly lost communications. Completely several times over several matches. During those times the field was actually behaving irradically, music timed wrong, events happening off queue (this was not a major competition thank goodness more a local expo).
We tried to use the DS (driver’s station) information but it wasn’t clear that it was representative that all messages sent were effectively received properly. What I’m looking for is a way to be sure from both the robot and the driver’s station that everything we are communicating is getting to where it should. Packet statistics and packet CRCs might help (UDP is obviously gonna need some form of tracking more than TCP which is reliable). I just am not confident of how all this information with what we already have confirms that everything sent to the robot is getting to the cRIO. Is there some documentation I can refer to on this?
We have experimented with logging back to the DS from the robot but we found that the additional traffic and the additional programming had several undesirable results. I’m sure that has application, but for some problems keeping the path to be written to a log short and as simple as possible is desirable. We looked earlier to see if we can access robot flash from Java. Can you point us in the direction of how we can learn more about that?
The Charts tab is new this year and was implemented to help shed light on situations like you describe without adding overhead to the robot code. All critical field communication is via UDP with CRCs. Video uses TCP/HTTP. With more insight into what this logging doesn’t capture, it is feasible that the default code or FRC_Communications may start logging additional data.
As a CSA, I’ve seen many robots that didn’t move or stopped moving, and the field is the easiest element to point to, but in my experience, it is rarely the issue – exceptions are damaged field cables and connectors and some late night debug sessions where zero robots could connect that required a reset of the access point. Hopefully the charts tab, especially if the code is instrumented, will help hilight the SW, radio, electrical, and other issues for quicker debugging and resolution. And in those rare situations when the field or field radio are to blame, I hope the charts tab on the various robots helps that debugging as well.
If anyone has a mysterious match that didn’t go well, I’ll be happy to review the log/tea leaves and see if we can shed some light. The log file viewer is in the Program Files/FRC Driver Station folder, and the logs are in the Public Users/Documents/FRC/Logs folder.
To log to flash from any of the languages, you should be able to use the file I/O APIs. I know this work on C++ and LV, and I’m pretty sure it works on Java, but I can’t verify it.
The TI Beagle Bone boots in under three seconds. It is a full Linux system inside. The thing I like is the Node.js using the Cloud9 IDE. But you could program in the language of your choice. $90.
The RN 121 is a WiFi device that has digital inputs and outputs, analog inputs and a UART for data streams. $40
The Bone isn’t going to do onboard video processing, on the other hand there are lots of better add in hardware that would be able to do that. Nice thing about the Bone someone smart could put a image processor on a “cape”.
If you want a few more CPU cycles then the Beagle Board is a better choice at about $180
I was just reading about the BeagleBone yesterday, and I thought I saw boot time of closer to one minute. I googled and found a pared down custom kernel boot time of 15 seconds. Do you have a source for the three second boot?
Since I work for NI, anything I say/ask can easily come off as defensive, but please don’t take it that way. As I said, I’m playing with one too, or at least I hope to this afternoon, and thus far it seems very cool. Without a cape, it seems like the I/O is pretty limited, and I’m a little concerned about burning it up if I short pins or reverse a cable, but it is a cool and capable little board.
In San Antonio, I spoke with 118, who are using one for image processing for this year’s game, so I don’t really understand you comments in that regard. Any experience to the contrary?
Guys,
When I started this question I was really intending that we start with a list of things we would want to see in the future. We should keep discussion to a minimum for the time being, please and just focus on the list. We can have discussions in another thread but between the two threads already talking on this subject, we will not be able to get a comprehensive list.
Thanks.
Al
