On the quality and complexity of software within FRC

[EricH]

Originally Posted by Pault

Yes, I completely realize that. I never said it was a bad thing that those threads exist. I was just pointing out how few threads actually provoke deeper conversations about coding than that. Imagine if nearly every thread about mechanical topics were "What CAD software should we use?", "Where can I learn to use X CAD software?", "Will this tank drive design function without breaking?", "How do gear ratios work?", etc. The technical side of CD would get boring really fast.

I did touch on some of that--later in the post.

Really simply, most teams' programmers get the robot really late in the season--and most teams' programmers are then trying to get the robot running under pressure from mechanical who wants drive time. Ugly code that works is of far, far greater value to the team than really nice, standards-compliant, reusable year-after-year code. That is the perception, at least. And it is very difficult to break out of that without a determined effort by one or more programmers to force the issue.

And then there's another problem. 4 years. Every 4 years, there is a complete turnover. (I intentionally exclude mentors from this.) Given that many teams have limited programmers in the first place (and, given some of those threads, it's tough to get a programmer to step up to replace a lone programmer who is moving on), there isn't really a good progression... so even if you do get a programmer or two who are starting to force the team towards high quality, or do more complex things, right about the time they hit that point they're gone, and someone else is starting from near-zero. You can argue that if they were doing it right that wouldn't be an issue, but getting to the point of doing it right can take YEARS.



As far as complexity...Let's just say that some mechanical mentors aren't willing to trust the programmers with more than basic sensors, and have to be poked, prodded, and otherwise convinced to (allow the programmers to) use the more advanced items. (I'm not one of them--but I prefer the simpler ways of doing things over the more complex ways.) OTOH, when the complex stuff works right, they suddenly want a lot more... Just got to get them there--again, that usually takes a programmer who insists on it until... oops, graduated another one, time to start over.

[Pault]

Originally Posted by Necroterra

Also, forbidding the modification of release code (that is, working code at competition) is a great idea. Aside from hotfixes / emergencies / changing something trivial like autonomous values, or controller inputs, messing with code at competition is bad practice.

I would like to introduce you to git. Version control and branches make it possible to do the stuff that you are talking about easily with almost no risk of shooting yourself in the foot. Just remember to commit often even in the rush of competition and never to send your robot into a match with untested code changes.

[Jared]

No one cares about how inefficient a routine is so long as it works in a match.

As somebody who honestly doesn't care how inefficient my (or anybody else's) software is so long as it works in a match, I hope I can explain my mentality.

The simple version is that FIRST is an engineering competition, not a science fair or research project. It's the difference between an engineer at a company that designs and manufactures really cheap CD players and a college professor. The engineer designing the CD player realizes that their solution isn't perfect, and shouldn't be, due to its limited parts cost and design time. The college professor has years and years to conduct research and strives to achieve perfection.

It's not always about doing things in a very academic, well documented, extremely optimized way, but instead it's about getting things done that work as well as they need to, but no better. Why shoot for 60 fps when 2 fps works just as well? Why learn about optimization techniques for things that are efficient enough? Why change code that already works when there is other work to be done?
To many, FIRST is an engineering competition. Teams must engineer a solution to a problem in a limited amount of time, just like real engineers do in the "real world". There is a finite amount of time to optimize an infinite number of things, so engineers must make tradeoffs. Should they give the programmers 10 hours with the robot to write vision software, or should they iterate on our hook so that we can lift totes easily? I didn't see a team this year that seriously benefited from having computer vision on their robot. Should a programmer spend time optimizing and threading their code when they could spend time practicing driving the robot?

Most people (myself included) judge robots by their ability to score points, prevent opponents from scoring, win matches (if applicable), seed high, and progress in the tournament structure of the competition.

A really cool mechanical thing that isn't very effective in a match is unimpressive - for instance, some 3D printed parts, many carbon fiber parts, many crazy magnesium alloy parts...

A really simple mechanical thing that is very effective in a match is very impressive. Look at the robots from 118, 254, 1114, and other really great teams. They use traditional methods of manufacturing with traditional materials because it works.

A really cool software thing that isn't very effective (vision processing this year) is unimpressive to the crowd.

A piece of software that just works is impressive, regardless of its efficiency, documentation, or organization.

I never saw a robot and thought "Gee, that team would be so much better off if there software was more efficient", or "that team could beat the Poofs if their vision system had a tiny bit less lag". Instead, I thought things like "if only their elevator was a little faster".

A lot of teams forbid the altering of code once it works, which is a disgusting practice. Teams that get 10-15 fps on a vision program and say "it's good enough" when they haven't even done their research on optimization techniques.

I have done both of these things. Once my code works, I don't change it. There are an infinite number of better things I can do with my time during build season than tweaking code to be slightly more efficient. If it's efficient enough to do what I need it to do, I don't want to waste either my time or my team's time with changing things.

What if 10-15 fps is enough? What if 1 frame per second is good enough?


I guess what I'm saying is that most teams software is 'good enough' and is rarely is what causes a team to be unsuccessful. There are many teams with very well written software and ineffective robots, but there aren't many teams with effective robots limited by bad software. Rarely are the best robots filled with extremely complex algorithms and parts, but rather, well engineered simple solutions. The code posted in Chief Delphi whitepapers often deals with concepts more advanced than anything found on any world champion robot.

[marshall]

Originally Posted by Jared

As somebody who honestly doesn't care how inefficient my (or anybody else's) software is so long as it works in a match, I hope I can explain my mentality.

The simple version is that FIRST is an engineering competition, not a science fair or research project. It's the difference between an engineer at a company that designs and manufactures really cheap CD players and a college professor. The engineer designing the CD player realizes that their solution isn't perfect, and shouldn't be, due to its limited parts cost and design time. The college professor has years and years to conduct research and strives to achieve perfection.

It's not always about doing things in a very academic, well documented, extremely optimized way, but instead it's about getting things done that work as well as they need to, but no better. Why shoot for 60 fps when 2 fps works just as well? Why learn about optimization techniques for things that are efficient enough? Why change code that already works when there is other work to be done?
To many, FIRST is an engineering competition. Teams must engineer a solution to a problem in a limited amount of time, just like real engineers do in the "real world". There is a finite amount of time to optimize an infinite number of things, so engineers must make tradeoffs. Should they give the programmers 10 hours with the robot to write vision software, or should they iterate on our hook so that we can lift totes easily? I didn't see a team this year that seriously benefited from having computer vision on their robot. Should a programmer spend time optimizing and threading their code when they could spend time practicing driving the robot?

Most people (myself included) judge robots by their ability to score points, prevent opponents from scoring, win matches (if applicable), seed high, and progress in the tournament structure of the competition.

A really cool mechanical thing that isn't very effective in a match is unimpressive - for instance, some 3D printed parts, many carbon fiber parts, many crazy magnesium alloy parts...

A really simple mechanical thing that is very effective in a match is very impressive. Look at the robots from 118, 254, 1114, and other really great teams. They use traditional methods of manufacturing with traditional materials because it works.

A really cool software thing that isn't very effective (vision processing this year) is unimpressive to the crowd.

A piece of software that just works is impressive, regardless of its efficiency, documentation, or organization.

I never saw a robot and thought "Gee, that team would be so much better off if there software was more efficient", or "that team could beat the Poofs if their vision system had a tiny bit less lag". Instead, I thought things like "if only their elevator was a little faster".





I have done both of these things. Once my code works, I don't change it. There are an infinite number of better things I can do with my time during build season than tweaking code to be slightly more efficient. If it's efficient enough to do what I need it to do, I don't want to waste either my time or my team's time with changing things.

What if 10-15 fps is enough? What if 1 frame per second is good enough?


I guess what I'm saying is that most teams software is 'good enough' and is rarely is what causes a team to be unsuccessful. There are many teams with very well written software and ineffective robots, but there aren't many teams with effective robots limited by bad software. Rarely are the best robots filled with extremely complex algorithms and parts, but rather, well engineered simple solutions. The code posted in Chief Delphi whitepapers often deals with concepts more advanced than anything found on any world champion robot.

+1.

I couldn't have said it better.

[Necroterra]

Originally Posted by Pault

I would like to introduce you to git. Version control and branches make it possible to do the stuff that you are talking about easily with almost no risk of shooting yourself in the foot. Just remember to commit often even in the rush of competition and never to send your robot into a match with untested code changes.

That, actually, is what I was trying to get at. I might have worded it badly, sorry.

The Gitflow workflow is, in my opinion, perfect for FRC Teams. Work on features and develop throughout build season, then once drive practice gets close, start a release branch, and put out a master update for the drivers to practice on (or of course just have them practice on develop... it depends on how much driver practice you would want to do). Then, a few days before each competition, start another release and clean code, disable any extraneous or nonworking features, decide on autonomous routines, etc. Changes at competition should be hotfix branch only.

[AlexanderTheOK]

Originally Posted by Jared

I didn't see a team this year that seriously benefited from having computer vision on their robot.

This part is really important here. I was a rookie in 2012, a year when vision processing was actually a useful tool for aiming due to the small size of the baskets. Every offseason afte,r I developed vision code (going from thresholds to SURF, ORB, and cascade classifiers) hoping that in the 3 YEARS I HAD LEFT, there would be ONE GAME in which vision processing was useful.

So far, the closest thing to useful vision processing in those 3 more years I spent programming has been cheesy vision. While it is an ingenious solution to an unexpected but consistent field failure, it is not robot vision processing, it was a simpler, quicker to develop, and more effective alternative to it.

This year, at my last regional as a student ever, Ventura, there were two teams that were able to consistently stack the 3 yellow totes, 1717 and 696.

Both robots accomplished this task without cameras, because cameras were not needed.

As others have said, it's an engineering competition, where we are making a robot that scores points. When code efficiency scores teams points, you'll see a lot of teams start making efficient code.

[John Retkowski]

I don't know why, but this thread reminds me of a joke me and one of the mechanical mentors started making this year about deciding to forget the code and just wind up the robot before every match. It's a mechanical team's dream...

[Basel A]

I agree with many of the above posters about code efficiency, but not about the relative gains of code improvements. While "getting it to work" is a bare minimum that most teams accept, automating small tasks can help a driver tremendously (place stack, shoot once in position), as can sophisticated control like maintaining lift position rather than open loop lift control by the driver. Like anything else there are diminishing returns, but I do think most teams would do better if they put a higher priority on their code/controls.

In my mind, there are two ways an individual can work to change the status quo of "no one cares about code." The first is obvious: work with your team to do what you believe is important. Different teams differently value scouting, CAD, marketing. There are teams that are known for their excellence in their preferred areas. Be the team that excels in code and spreads the word.

The second way is also obvious. If you think people should critique code on Chief Delphi the same way they do CAD, then you do that. It only takes two people to have a conversation. Perhaps you'll start a larger trend, maybe you'll just give some good advice. Teams quite often post their codebases on CD, and usually receive no response. Start the conversation you want to see.

I do think FRC would better prepare students for 21st century engineering if FRC games placed a higher value on code, but I also recognise the difficulty of effectively doing that while maintaining watchability and other priorities. Not to mention that it's not FIRST's chief goal to prepare students but rather to inspire them.

[Ari423]

Limited robot access: It is inherently difficult to develop software for a machine that isn't built yet...

Time limitation: six weeks is a really short timeframe for software...

What I find helps is writing the code before the robot is built. Sure you end up modifying or deleting most of it, but taking the time to make things looks nice while you are waiting for the robot to be ready to be tested gives you the standard for the modifications you make to your code. Also, thinking about all of the different possibilities for a general idea opens your mind to new ideas the mechanical team hadn't thought of.

Not enough manpower: I think that many teams have only a single programmer. If so, it is not trivial for one person without any support to learn and apply everything that goes into buiilding a good project.

Not enough support: I also wouldn't be surprised if many teams had no dedicated software mentors.

I understand about what people are saying about only having one programmer and no programming mentors, but I am in that situation as well and I still am able to make efficient and well-formatted code. I think what is important is having someone else's clean code to look up to. The programmer before me left me his old code. While some of it was dirty, most of the competition code was neat and easy to understand, even without comments. Perhaps it's because he was an artist as well, but I really appreciated the time he took to make the code look nice (which helped debugging and such because you knew instantly what you were looking at).

Not enough incentive for good software: aside from innovation in controls, there is very little incentive for medium or low performing teams to put a lot of resources into their code....The roboRIO is more than fast enough for pretty much everybody, and with the CAN Talons, the vast majority of functionality is achieved by very simple code.

While I agree that there isn't much incentive in FIRST, I feel that programmers should strive for more than functionality. When you leave FIRST and go to get a job, your employers will want other programmers to be able to understand your code, and bad practices and dirty code don't help with this. When your code is clean it makes updating it and advancing it much easier.

Most teams' programmers get the robot really late in the season--and most teams' programmers are then trying to get the robot running under pressure from mechanical who wants drive time. Ugly code that works is of far, far greater value to the team than really nice, standards-compliant, reusable year-after-year code.

This, again, is where making code before you get the robot helps. You will need to rewrite stuff, but it's less work than writing the entire code from scratch.

To many, FIRST is an engineering competition. Teams must engineer a solution to a problem in a limited amount of time, just like real engineers do in the "real world". There is a finite amount of time to optimize an infinite number of things, so engineers must make tradeoffs.

I agree with what you are saying; there is a point where code can become over-developed. But there are some practices that take no extra time, but make the code more efficient and easier to read. And if it doesn't work the first time, it makes it easier for the programmer or someone helping to debug. Learning good coding practices in the off-season can greatly help during the build season.

It's an engineering competition, where we are making a robot that scores points. When code efficiency scores teams points, you'll see a lot of teams start making efficient code.

Even though optimized code doesn't score you points, it does make getting to the point where you can score points and debugging when you can't score points easier. It takes less time to copy to the robot and a well-formatted dashboard can help you debug the robot mid-match and save yourself half a match (which we needed to do and we able to thanks to our dashboard).

[Brian Maher]

Originally Posted by connor.worley

Requiring teams to submit their code to the judges for review might fix this, but I don't know if the volunteer capacity exists.

I think this has a lot of potential as an optional submission, but require it to be considered for the Innovation in Control Award, like how a Business Plan submission is required for the Entrepreneurship Award.

This year, I had our team's programmers assemble a document including the code, with full comments, and some flowcharts and rationales. It helped them better understand their decision making process, realize its advantages and weaknesses, and identify ways to improve. It's also nice to have another deliverable for judges.

P.S. getting programmers to document their work can be difficult.

[sforsyth]

You most likely have not worked in the real world... ahhhh to have a fresh college mind where everything is perfect and there is plenty of time to make everything perfect and just the way *I* want it

There are others that have said it in this thread I'm sure... the perfect code is whatever you think it is, your perfect code is not necessarily mine. If it works, then it is probably perfect to the person that wrote it.

Most of these kids are brand new to programming and it is better served just to learn the language, leave the perfection to College courses if that is what they decide to go into.

That being said, I'm sure there there are many ways to help them learn to write clean code and that really should be what you are asking... how can we help the kids?

[jman4747]

Originally Posted by faust1706

A lot of teams forbid the altering of code once it works, which is a disgusting practice.

I would say that depends on the timing. If you're 10 minutes from queuing for another match or in the finals and everything is working than I'd usually say leave it. If it's mid August then yea have at it. With what we need the code to do and for how long it's not always smart to change a stable iteration. If it's not critical that the robot be working for a while then optimization should obviously be encouraged. If in the middle of competition however I wouldn't be making major changes to algorithms, structure, or data flow etc etc.

[AdamHeard]

Aside from vision (and possibly elaborate real time spline generation) what even deals with enough data in FRC to worry about O complexity?

[SoftwareBug2.0]

Originally Posted by faust1706

Year after year, the vast majority of teams are writing poorly written, inefficient, and disorganized code. Their code base fails to generalize for multiple years due to poor design, but no one seems to care or talk about this.

Someone posts a cad drawing of a robot that they did for practice. People ask questions asking about the foundations of the design ("How thick is the G10?"), but no one asks questions when code is released. No one asks, "what is the *big-O notation of this function?" No one cares enough to truly scrutinize someone else's code. No one cares that an on board vision program isn't threaded. No one cares about how inefficient a routine is so long as it works in a match. A lot of teams forbid the altering of code once it works, which is a disgusting practice. Teams that get 10-15 fps on a vision program and say "it's good enough" when they haven't even done their research on optimization techniques. The vast majority of code would get a 'C' at best in an intro to programming class in a high school. So why is no one talking about this?

I agree there's a lot of poor quality code in FRC and little higher-level discussion. I think there are many reasons for this, some of which have already been mentioned.

-A lot of high schoolers do poorly with programming: Even though the AP computer science test is ridiculously easy over 30% of students who took it last year received a 1 (for those not familiar with AP tests, 1 is the lowest possible grade). And out of all the AP tests given that was the highest percentage of ones awarded.

-Reading code is not very fun. I know you've released a couple of interesting projects and it's cool to know what they do but I'm pretty much not going to read their source code unless I want to reuse them.

-The code libraries that you're given in FRC don't set you up to do fancy things well. It's non-modular in a way that makes unit testing hard. It's hard enough that I don't think I would have figured out a way to do it when I was in high school. Sure there are teams that have worked around this and built things that you could build on, but they're not well known. For example, my team publishes our code which includes a very different way of using C++ but I don't know of any other team that's even looked closely at it. Similarly, 1510 has published an alternate framework for Java which they've been using but I only know of one other team that has bothered to try it.

-Optimization and efficiency are ignored because they can be. To do what the average team does the processor is about two orders of magnitude faster than it needs to be and you have 3 or 4 orders of magnitude more RAM than needed. I always think that I'm going to get to teaching the students on my team about efficiency when we get to the point where there would be some benefit to doing so, but things always end up 'good enough'.

-FRC games allow success with limited software functionality. Every year there are robots that have no autonomous program at all and win tournaments. This is silly. The game should start with each robot in a box taped on the floor and you shouldn't get driver control until it leaves the box.

One thing that I must disagree with you about is that restricting changes to code can be rational. Even if your code is correct it does not follow that your system will behave as expected. Minor code changes can and do trigger bugs in other parts of the system. You might now be triggering a different (and buggy) code path in a library you're using. We've all seen compiler bugs before. And there's the famous "download silently failed and left system in an unusable state". When one of these other problems occurs it would be nice to find out why but right before a match is neither the time nor place.

Also, there is some interesting coding going on even if it's not going on a robot. For example, here: http://www.chiefdelphi.com/forums/sh...hreadid=137451.

[gblake]

I am surprised by how few people have challenged the OP's assertion that I'll paraphrase as "most FRC software stinks"; and by how many respondents seem to embrace that assessment.

It is completely impossible to make a judgment like that without knowing what the software is required/expected to do.

You can easily make a case for McCall's criteria or other rather abstract notions of what is/isn't high-quality software being completely irrelevant if the software is satisfying all requirements placed on it.

McCall, et al published their important work to take a shot at being able evaluate software written for use in a certain general context. However, outside of that context, other criteria often apply.

First agree on what you want a given collection of software to do and be. Then have a proper conversation about quality. Don't put the cart before the horse. Don't let the tail wag the dog.

Blake