On the quality and complexity of software within FRC

[connor.worley]

Originally Posted by faust1706

Another idea we had with nvidia is to have an award for quality of source code, teams that want to be a contender would submit their code online before competition began.

If you do this, allow submissions after competition. Lots of cool features get added between competitions.

[Michael Hill]

Originally Posted by faust1706

Sorry for lack of quotes, on mobile. The blame is also on us. My mentor had me calculate the big O of every algorithm I designed, and he questioned every algorithm design I had. If I didn't have a well thought out answer, I wasn't allowed to use it. We've been using that code base ever since, haven't changed a line of it, and it has been able to adapt to the 2013 game, a 3 camera system in 2014, and 3d imaging this year.

Part of our partnership with nvidia will be promoting good code in the community and getting people excited about programming. We thought of the idea of hosting a class at worlds, but we feel even if we had it, not a lot of people would show up.

I do not have a solution to this deeply rooted problem as it ultimately comes down to each individual team. Addressing it as a problem is the first step. Another idea we had with nvidia is to have an award for quality of source code, teams that want to be a contender would submit their code online before competition began.

Here's one of the interesting things about FRC. Your mentor has taught you some really good skills that are applicable at a much higher level than what's normally done in FRC. A bit academic, but from your other posts I've seen on here, it looks like it's made you a better developer. While code may have been developed faster for the robot, it doesn't really matter because you learned something. While you may not calculate the Big O for every algorithm you do, you will probably also keep it in mind in any future algorithms your write.

[teslalab2]

I can say this, I was the programming team and half of the mechanical team, and though I spent a lot of time on the code, it worked and that was about it... when I wanted to finally get around to fixing the spaghetti in my code, the robot mechanically suicided and I didn't have time to fix my code, this went on constantly until build was over.

[Necroterra]

Throwing in my 2 cents about some of the reasons teams might have poor software.

• Not enough manpower: I think that many teams have only a single programmer. If so, it is not trivial for one person without any support to learn and apply everything that goes into buiilding a good project.
• Not enough support: I also wouldn't be surprised if many teams had no dedicated software mentors. Additionally, even here on CD we don't have much of a focus on software, and the WPILIB documentation, while decent, doesn't teach anything about how to design good software (Git, how to organize code, patterns, etc.)
• Not enough visibility: Almost every other aspect of a team is immediately present in their robot and team - you can see the mechanical design, the team branding, the manufacturing, etc. but aside from the occasional code release on CD, I would bet that many teams' programmers haven't really even looked at anyone else's code.
• Not enough incentive for good software: aside from innovation in controls, there is very little incentive for medium or low performing teams to put a lot of resources into their code. Most teams don't have a robot that performs well enough for things like extendable/flexible code or vision to actually matter. The roboRIO is more than fast enough for pretty much everybody, and with the CANTalons, the vast majority of functionality is achieved by very simple code.
• Team's attitude towards software: I think some teams have a very negative or accusatory attitude towards the software subteam. I've definitely seen some shirts / signatures / quotes / jokes that seem to indicate some (probably not conscious) lack of respect towards programmers. I think our team had some bad experiences with these sorts of things before I joined, but I can't myself comment on it.
• Limited robot access: It is inherently difficult to develop software for a machine that isn't built yet. Design/fabrication teams will always want to improve the robot for all six weeks, meaning the programmers have to fight for time with the robot.
• Time limitation: six weeks is a really short timeframe for software. Implementing a good workflow, doing code reviews / refactors, etc. just doesn't really make sense on such a short timescale. I think many teams don't do these sorts of things during the offseason either, either because they just don't work much on robots outside of the main season or for any of the reasons above.

I just think overall, the structure and culture of FRC isn't honestly that great for learning software unless the students are really motivated. I've definitely gotten a lot out of it, but I also have been doing a lot of research and outside work. In my one year of experience, I ran into many of the above issues.

Also, forbidding the modification of release code (that is, working code at competition) is a great idea. Aside from hotfixes / emergencies / changing something trivial like autonomous values, or controller inputs, messing with code at competition is bad practice.

[Pault]

Originally Posted by Necroterra

Also, forbidding the modification of release code (that is, working code at competition) is a great idea. Aside from hotfixes / emergencies / changing something trivial like autonomous values, or controller inputs, messing with code at competition is bad practice.

I would like to introduce you to git. Version control and branches make it possible to do the stuff that you are talking about easily with almost no risk of shooting yourself in the foot. Just remember to commit often even in the rush of competition and never to send your robot into a match with untested code changes.

[Necroterra]

Originally Posted by Pault

I would like to introduce you to git. Version control and branches make it possible to do the stuff that you are talking about easily with almost no risk of shooting yourself in the foot. Just remember to commit often even in the rush of competition and never to send your robot into a match with untested code changes.

That, actually, is what I was trying to get at. I might have worded it badly, sorry.

The Gitflow workflow is, in my opinion, perfect for FRC Teams. Work on features and develop throughout build season, then once drive practice gets close, start a release branch, and put out a master update for the drivers to practice on (or of course just have them practice on develop... it depends on how much driver practice you would want to do). Then, a few days before each competition, start another release and clean code, disable any extraneous or nonworking features, decide on autonomous routines, etc. Changes at competition should be hotfix branch only.

[Ari423]

Limited robot access: It is inherently difficult to develop software for a machine that isn't built yet...

Time limitation: six weeks is a really short timeframe for software...

What I find helps is writing the code before the robot is built. Sure you end up modifying or deleting most of it, but taking the time to make things looks nice while you are waiting for the robot to be ready to be tested gives you the standard for the modifications you make to your code. Also, thinking about all of the different possibilities for a general idea opens your mind to new ideas the mechanical team hadn't thought of.

Not enough manpower: I think that many teams have only a single programmer. If so, it is not trivial for one person without any support to learn and apply everything that goes into buiilding a good project.

Not enough support: I also wouldn't be surprised if many teams had no dedicated software mentors.

I understand about what people are saying about only having one programmer and no programming mentors, but I am in that situation as well and I still am able to make efficient and well-formatted code. I think what is important is having someone else's clean code to look up to. The programmer before me left me his old code. While some of it was dirty, most of the competition code was neat and easy to understand, even without comments. Perhaps it's because he was an artist as well, but I really appreciated the time he took to make the code look nice (which helped debugging and such because you knew instantly what you were looking at).

Not enough incentive for good software: aside from innovation in controls, there is very little incentive for medium or low performing teams to put a lot of resources into their code....The roboRIO is more than fast enough for pretty much everybody, and with the CAN Talons, the vast majority of functionality is achieved by very simple code.

While I agree that there isn't much incentive in FIRST, I feel that programmers should strive for more than functionality. When you leave FIRST and go to get a job, your employers will want other programmers to be able to understand your code, and bad practices and dirty code don't help with this. When your code is clean it makes updating it and advancing it much easier.

Most teams' programmers get the robot really late in the season--and most teams' programmers are then trying to get the robot running under pressure from mechanical who wants drive time. Ugly code that works is of far, far greater value to the team than really nice, standards-compliant, reusable year-after-year code.

This, again, is where making code before you get the robot helps. You will need to rewrite stuff, but it's less work than writing the entire code from scratch.

To many, FIRST is an engineering competition. Teams must engineer a solution to a problem in a limited amount of time, just like real engineers do in the "real world". There is a finite amount of time to optimize an infinite number of things, so engineers must make tradeoffs.

I agree with what you are saying; there is a point where code can become over-developed. But there are some practices that take no extra time, but make the code more efficient and easier to read. And if it doesn't work the first time, it makes it easier for the programmer or someone helping to debug. Learning good coding practices in the off-season can greatly help during the build season.

It's an engineering competition, where we are making a robot that scores points. When code efficiency scores teams points, you'll see a lot of teams start making efficient code.

Even though optimized code doesn't score you points, it does make getting to the point where you can score points and debugging when you can't score points easier. It takes less time to copy to the robot and a well-formatted dashboard can help you debug the robot mid-match and save yourself half a match (which we needed to do and we able to thanks to our dashboard).

[Jared]

No one cares about how inefficient a routine is so long as it works in a match.

As somebody who honestly doesn't care how inefficient my (or anybody else's) software is so long as it works in a match, I hope I can explain my mentality.

The simple version is that FIRST is an engineering competition, not a science fair or research project. It's the difference between an engineer at a company that designs and manufactures really cheap CD players and a college professor. The engineer designing the CD player realizes that their solution isn't perfect, and shouldn't be, due to its limited parts cost and design time. The college professor has years and years to conduct research and strives to achieve perfection.

It's not always about doing things in a very academic, well documented, extremely optimized way, but instead it's about getting things done that work as well as they need to, but no better. Why shoot for 60 fps when 2 fps works just as well? Why learn about optimization techniques for things that are efficient enough? Why change code that already works when there is other work to be done?
To many, FIRST is an engineering competition. Teams must engineer a solution to a problem in a limited amount of time, just like real engineers do in the "real world". There is a finite amount of time to optimize an infinite number of things, so engineers must make tradeoffs. Should they give the programmers 10 hours with the robot to write vision software, or should they iterate on our hook so that we can lift totes easily? I didn't see a team this year that seriously benefited from having computer vision on their robot. Should a programmer spend time optimizing and threading their code when they could spend time practicing driving the robot?

Most people (myself included) judge robots by their ability to score points, prevent opponents from scoring, win matches (if applicable), seed high, and progress in the tournament structure of the competition.

A really cool mechanical thing that isn't very effective in a match is unimpressive - for instance, some 3D printed parts, many carbon fiber parts, many crazy magnesium alloy parts...

A really simple mechanical thing that is very effective in a match is very impressive. Look at the robots from 118, 254, 1114, and other really great teams. They use traditional methods of manufacturing with traditional materials because it works.

A really cool software thing that isn't very effective (vision processing this year) is unimpressive to the crowd.

A piece of software that just works is impressive, regardless of its efficiency, documentation, or organization.

I never saw a robot and thought "Gee, that team would be so much better off if there software was more efficient", or "that team could beat the Poofs if their vision system had a tiny bit less lag". Instead, I thought things like "if only their elevator was a little faster".

A lot of teams forbid the altering of code once it works, which is a disgusting practice. Teams that get 10-15 fps on a vision program and say "it's good enough" when they haven't even done their research on optimization techniques.

I have done both of these things. Once my code works, I don't change it. There are an infinite number of better things I can do with my time during build season than tweaking code to be slightly more efficient. If it's efficient enough to do what I need it to do, I don't want to waste either my time or my team's time with changing things.

What if 10-15 fps is enough? What if 1 frame per second is good enough?


I guess what I'm saying is that most teams software is 'good enough' and is rarely is what causes a team to be unsuccessful. There are many teams with very well written software and ineffective robots, but there aren't many teams with effective robots limited by bad software. Rarely are the best robots filled with extremely complex algorithms and parts, but rather, well engineered simple solutions. The code posted in Chief Delphi whitepapers often deals with concepts more advanced than anything found on any world champion robot.

[marshall]

Originally Posted by Jared

As somebody who honestly doesn't care how inefficient my (or anybody else's) software is so long as it works in a match, I hope I can explain my mentality.

The simple version is that FIRST is an engineering competition, not a science fair or research project. It's the difference between an engineer at a company that designs and manufactures really cheap CD players and a college professor. The engineer designing the CD player realizes that their solution isn't perfect, and shouldn't be, due to its limited parts cost and design time. The college professor has years and years to conduct research and strives to achieve perfection.

It's not always about doing things in a very academic, well documented, extremely optimized way, but instead it's about getting things done that work as well as they need to, but no better. Why shoot for 60 fps when 2 fps works just as well? Why learn about optimization techniques for things that are efficient enough? Why change code that already works when there is other work to be done?
To many, FIRST is an engineering competition. Teams must engineer a solution to a problem in a limited amount of time, just like real engineers do in the "real world". There is a finite amount of time to optimize an infinite number of things, so engineers must make tradeoffs. Should they give the programmers 10 hours with the robot to write vision software, or should they iterate on our hook so that we can lift totes easily? I didn't see a team this year that seriously benefited from having computer vision on their robot. Should a programmer spend time optimizing and threading their code when they could spend time practicing driving the robot?

Most people (myself included) judge robots by their ability to score points, prevent opponents from scoring, win matches (if applicable), seed high, and progress in the tournament structure of the competition.

A really cool mechanical thing that isn't very effective in a match is unimpressive - for instance, some 3D printed parts, many carbon fiber parts, many crazy magnesium alloy parts...

A really simple mechanical thing that is very effective in a match is very impressive. Look at the robots from 118, 254, 1114, and other really great teams. They use traditional methods of manufacturing with traditional materials because it works.

A really cool software thing that isn't very effective (vision processing this year) is unimpressive to the crowd.

A piece of software that just works is impressive, regardless of its efficiency, documentation, or organization.

I never saw a robot and thought "Gee, that team would be so much better off if there software was more efficient", or "that team could beat the Poofs if their vision system had a tiny bit less lag". Instead, I thought things like "if only their elevator was a little faster".





I have done both of these things. Once my code works, I don't change it. There are an infinite number of better things I can do with my time during build season than tweaking code to be slightly more efficient. If it's efficient enough to do what I need it to do, I don't want to waste either my time or my team's time with changing things.

What if 10-15 fps is enough? What if 1 frame per second is good enough?


I guess what I'm saying is that most teams software is 'good enough' and is rarely is what causes a team to be unsuccessful. There are many teams with very well written software and ineffective robots, but there aren't many teams with effective robots limited by bad software. Rarely are the best robots filled with extremely complex algorithms and parts, but rather, well engineered simple solutions. The code posted in Chief Delphi whitepapers often deals with concepts more advanced than anything found on any world champion robot.

+1.

I couldn't have said it better.

[AlexanderTheOK]

Originally Posted by Jared

I didn't see a team this year that seriously benefited from having computer vision on their robot.

This part is really important here. I was a rookie in 2012, a year when vision processing was actually a useful tool for aiming due to the small size of the baskets. Every offseason afte,r I developed vision code (going from thresholds to SURF, ORB, and cascade classifiers) hoping that in the 3 YEARS I HAD LEFT, there would be ONE GAME in which vision processing was useful.

So far, the closest thing to useful vision processing in those 3 more years I spent programming has been cheesy vision. While it is an ingenious solution to an unexpected but consistent field failure, it is not robot vision processing, it was a simpler, quicker to develop, and more effective alternative to it.

This year, at my last regional as a student ever, Ventura, there were two teams that were able to consistently stack the 3 yellow totes, 1717 and 696.

Both robots accomplished this task without cameras, because cameras were not needed.

As others have said, it's an engineering competition, where we are making a robot that scores points. When code efficiency scores teams points, you'll see a lot of teams start making efficient code.

[John Retkowski]

I don't know why, but this thread reminds me of a joke me and one of the mechanical mentors started making this year about deciding to forget the code and just wind up the robot before every match. It's a mechanical team's dream...

[Basel A]

I agree with many of the above posters about code efficiency, but not about the relative gains of code improvements. While "getting it to work" is a bare minimum that most teams accept, automating small tasks can help a driver tremendously (place stack, shoot once in position), as can sophisticated control like maintaining lift position rather than open loop lift control by the driver. Like anything else there are diminishing returns, but I do think most teams would do better if they put a higher priority on their code/controls.

In my mind, there are two ways an individual can work to change the status quo of "no one cares about code." The first is obvious: work with your team to do what you believe is important. Different teams differently value scouting, CAD, marketing. There are teams that are known for their excellence in their preferred areas. Be the team that excels in code and spreads the word.

The second way is also obvious. If you think people should critique code on Chief Delphi the same way they do CAD, then you do that. It only takes two people to have a conversation. Perhaps you'll start a larger trend, maybe you'll just give some good advice. Teams quite often post their codebases on CD, and usually receive no response. Start the conversation you want to see.

I do think FRC would better prepare students for 21st century engineering if FRC games placed a higher value on code, but I also recognise the difficulty of effectively doing that while maintaining watchability and other priorities. Not to mention that it's not FIRST's chief goal to prepare students but rather to inspire them.

[EricH]

Originally Posted by Pault

All of the threads are something like either "What programming language should I use?", "Where do I go to learn how to code in X language?", "Why does this code not work?", "What is a PID?", etc. I think it could really benefit the teams on here if there was more in depth conversation.

Caution: "Not a Programmer" Talking in Programming Thread.

I'm going to take the above as kind of a "trigger point". Mainly because, it's what got my attention.

Who is asking those questions?

No, seriously, who is asking those questions? And, just for grins, why?

Think about it.


Now that you've thought about it, from what I've seen an awful lot of those threads are your first-time programmers. Not 2nd, 3rd, or 4th year programmers, but first-timers. The exception is the "why is this not working", which can come from anybody who is trying something new, or just wants a code review.

The other factor is that many teams don't give their programmers a robot until Week 5.75 if the programmers are lucky. Then they expect the robot to work in the first match on the field. See: hotfix the code and pray it works. Then fix some other bug. Repeat.


1197 does have a good programming team--but I know enough not to ask questions. (I leave that to the programming mentors.)

[Pault]

Originally Posted by EricH

Caution: "Not a Programmer" Talking in Programming Thread.

I'm going to take the above as kind of a "trigger point". Mainly because, it's what got my attention.

Who is asking those questions?

No, seriously, who is asking those questions? And, just for grins, why?

Think about it.


Now that you've thought about it, from what I've seen an awful lot of those threads are your first-time programmers. Not 2nd, 3rd, or 4th year programmers, but first-timers. The exception is the "why is this not working", which can come from anybody who is trying something new, or just wants a code review.

The other factor is that many teams don't give their programmers a robot until Week 5.75 if the programmers are lucky. Then they expect the robot to work in the first match on the field. See: hotfix the code and pray it works. Then fix some other bug. Repeat.


1197 does have a good programming team--but I know enough not to ask questions. (I leave that to the programming mentors.)

Yes, I completely realize that. I never said it was a bad thing that those threads exist. I was just pointing out how few threads actually provoke deeper conversations about coding than that. Imagine if nearly every thread about mechanical topics were "What CAD software should we use?", "Where can I learn to use X CAD software?", "Will this tank drive design function without breaking?", "How do gear ratios work?", etc. The technical side of CD would get boring really fast.

[EricH]

Originally Posted by Pault

Yes, I completely realize that. I never said it was a bad thing that those threads exist. I was just pointing out how few threads actually provoke deeper conversations about coding than that. Imagine if nearly every thread about mechanical topics were "What CAD software should we use?", "Where can I learn to use X CAD software?", "Will this tank drive design function without breaking?", "How do gear ratios work?", etc. The technical side of CD would get boring really fast.

I did touch on some of that--later in the post.

Really simply, most teams' programmers get the robot really late in the season--and most teams' programmers are then trying to get the robot running under pressure from mechanical who wants drive time. Ugly code that works is of far, far greater value to the team than really nice, standards-compliant, reusable year-after-year code. That is the perception, at least. And it is very difficult to break out of that without a determined effort by one or more programmers to force the issue.

And then there's another problem. 4 years. Every 4 years, there is a complete turnover. (I intentionally exclude mentors from this.) Given that many teams have limited programmers in the first place (and, given some of those threads, it's tough to get a programmer to step up to replace a lone programmer who is moving on), there isn't really a good progression... so even if you do get a programmer or two who are starting to force the team towards high quality, or do more complex things, right about the time they hit that point they're gone, and someone else is starting from near-zero. You can argue that if they were doing it right that wouldn't be an issue, but getting to the point of doing it right can take YEARS.



As far as complexity...Let's just say that some mechanical mentors aren't willing to trust the programmers with more than basic sensors, and have to be poked, prodded, and otherwise convinced to (allow the programmers to) use the more advanced items. (I'm not one of them--but I prefer the simpler ways of doing things over the more complex ways.) OTOH, when the complex stuff works right, they suddenly want a lot more... Just got to get them there--again, that usually takes a programmer who insists on it until... oops, graduated another one, time to start over.