A Vision Program that teaches itself the game

[sparkytwd]

Originally Posted by yash101

I think what I will do is calculate the angle between each consecutive point. I will change the position and angle constantly. The gyro will be used for accurate direction measurements. Whenever the robot is looking at the goal, the gyro will be recalibrated, yielding max-accuracy!

Sensor Fusion

[StevenB]

You might want to read some of the RoboCup literature - just do a Google Scholar search for "RoboCup". At this point, the wheeled-robot classes are actually pretty decent, and the small class is very good. The RoboCup community has solved a lot of interesting problems along the way. But just to give a little perspective: their game is very simple, and the teams are groups of PhD students who have been working on the robots for several years.

When I was taking a machine learning class and we were choosing course projects, the professor suggested a simple benchmark to tell if the project was appropriate: Is the task easy for a human? If not, it's probably between hard and impossible for a computer, unless it involves huge amounts of data or extremely fast reaction times. Driving an FRC robot is hard, way harder than driving a car.

Here's an interesting place to start that's a little easier: Given a video that shows the whole field for the duration of an Aerial Assist match, calculate the score. Don't worry about penalties. Just track the robots, track the balls (there's only 2!), and keep track of the score.

Once you can do something like this, you will have solved a number of the hard vision and analysis tasks, and will be in a position to make a robot react to play the game. Also, this would be a ludicrously awesome scouting tool.

[yash101]

The reason why it seems so difficult to automate many things on the robot is because we are thinking about the big picture. After you decompose the robot behavior down into small pieces, you get things that don't actually seem too difficult to implement! When you have tools such as Octave and MatLab, things like AI and ML become like, "How in the world did I implement this uber-complicated thing in just 3 lines of code?!". It is quite simple to use OpenCV and acquire data, such as target, robot position, etc.

Just think of this. For the 2014 game, if you just acquired the ball and shot it into the goal -- no passing, etc., you could break up the game into the following parts:
-shooting -- optimal position, shooting power, robot velocity, robot acceleration
-driving -- optimal speed, obstacle avoidance, static field, dynamic field, etc.
-picking up -- map of the gamepieces

This all can be accomplished by quite simple OpenCV + MatLab code!

Now, let's try a practice implementation (idea, not code):

OPENCV: generate field view, find all obstacles, find all the gamepieces, triangulate robot position, gather facing direction <-- this can be done in the matter of 600 lines of OpenCV code.
INPUT: wait for user input -- what to do next? drive to a location, shoot, or intake?
PERFORM:
  DRIVE: field data would be sent to the MatLab generated code.
  SHOOT: find the optimal shooting location, use DRIVE to prepare the shot and go for it
  INTAKE: read the list of gamepieces. Use DRIVE to get to the gamepiece. Use native code to pick up

These are all things that teams are already doing. However, I don't know of any team that has tried to perform it all.

Now, the program is capable of playing the game almost by itself -- little but some human intervention.

Now say that the driver wants to relax and doesn't want to drive by themselves -- sure, get new drivers , but even better would be if you program a bit of AI so the robot can play by itself.

Forget about the DRIVE feature, above. Now, the list is -- INTAKE, SHOOT, and maybe a WAIT command.

This is what I would do:

Check if the robot is loaded. If it is ready, go directly into SHOOT. Otherwise, WAIT (do nothing, but watch and wait for an event) for a gamepiece to be available. INTAKE the gamepiece. Use SHOOT to score. Repeat this over and over again until the game timer is up.

^^ You have a fully autonomous robot. Sure, it won't be as competitive as a human -- it couldn't defend or pass, or anything, but a computer can think very fast to a very high accuracy. This means that a computer can perform all of this with a higher accuracy than a human could!

The code would of course be much more difficult if you wanted to implement features like passing or catching. And even better is that there can be human control overrides, so if the human wants the robot to do something his way, it can do it his way.

[JamesTerm]

Originally Posted by yash101

The reason why it seems so difficult to automate many things on the robot is because we are thinking about the big picture. After you decompose the robot behavior down into small pieces, you get things that don't actually seem too difficult to implement!

Writing a goals class and breaking down the big picture to smaller pieces is the easy part. The devil is in the details at the bottom end.

[MatthewC529]

Originally Posted by JamesTerm

Writing a goals class and breaking down the big picture to smaller pieces is the easy part. The devil is in the details at the bottom end.

Exactly. We know it isn't impossible. It's just the feasibility I personally feel is being downplayed. The feasibility of creating a robot that adapts to a given game in a meaningful way I think requires you to be able to look at it as more than a little OpenCV and clever data gathering. The details are important. Everything seems simple in the big picture until you break it down realtime. Though you are not wrong in saying AI/ML can be deceptively simple.

You think of this as a game in code because it is a game. It's why I use iterative. Reminiscent of a game loop. But unlike a game it's a lot more difficult to handle AI and ML in reality, let alone competition.

Sorry for spelling errors. Mobile is not friendly to my hands.

[StevenB]

So, I'm a graduate student in electrical engineering, and have taken classes in machine learning, artificial intelligence, image processing, and computer vision. I've spent four summers in internships doing image processing with MATLAB and OpenCV.


Ok, I hesitate to call myself an expert, but I do know what I'm talking about.

Originally Posted by yash101

When you have tools such as Octave and MatLab, things like AI and ML become like, "How in the world did I implement this uber-complicated thing in just 3 lines of code?!". It is quite simple to use OpenCV and acquire data, such as target, robot position, etc.

I love MATLAB, and yes, it does let you do some pretty neat things with very little code. You can develop and prototype algorithms very quickly, and there seems to be a function for everything you want to do. I can't speak quite as highly of OpenCV, but it too allows you to quickly piece together a lot of computer vision building blocks to build new things. There are dozens of other great libraries out there too - stuff like ROS, VLFeat, PyML, and more.

But even with such great tools, I think you are radically underestimating the problem. Seriously, if you create a robot that can play Aerial Assist autonomously the way you're describing, you will have done enough innovative work to publish several papers and get a PhD.

I'm not saying this to discourage you. You've got a lot of passion and excitement, and a lot of great ideas, and I want to see something cool and innovative come out of it. You can do great work, and I look forward to seeing it. But I encourage you to focus your effort on a small piece of the problem. It may feel like a teeny, tiny, insignificant part. But unless you start with something small, it's hard to finish. I'm a dreamer by nature - like you, I have big ideas and grand plans, and I like to start new things. But too many of my projects have died shortly after I wrote down the grand vision, because I took on too much at once.

So, think about a small part of the problem, but also something you can get excited about.

• Program your robot to catch a ball thrown to it by a human. Use an onboard camera (looks like you've already got one on your bot) to track the trajectory of the ball, and have the robot drive into position to grab it.
• Place a series of traffic cones (or similar large bright objects) on your playing field. Make the robot chase down the ball and pick it up without knocking over any cones.
• Program your robot to automatically score the ball after picking it up a random distance from the goal, while avoiding traffic cones.
• Build the multi-camera system you described, and program the robot to play defense. A robot with red bumpers is trying to pick up a red ball. Your job is to stay between the red robot and the red ball.

These are all things that are doable in a summer, but to my knowledge, no team has done these. Pick one of these, or something similar, break it down into tiny tasks, and see what you can do. Ask questions, read papers, and write some code. With enough dedication, you'll come up with something really great, and I look forward to seeing it!

Originally Posted by yash101

This all can be accomplished by quite simple OpenCV + MatLab code!

Someone should go tell the RoboCup teams. They've been working on this for 15+ years.

[yash101]

Originally Posted by JamesTerm

Writing a goals class and breaking down the big picture to smaller pieces is the easy part. The devil is in the details at the bottom end.

Well, y'all are the experts, so I'd have to trust you on that. However, I have come up with quite a tangible and possible plan. The entire program will basically be split up into parts. Also, it will only have basic offense features. I a not thinking about making the entire thing pure AI/ML. A great portion will be basic IF's and ELSE's, where if the ball is ahead, go ahead, otherwise go backwards, etc. There is only one way where I think it would be efficient to use Machine Learning -- Best shooting spot, saying that the game is one in which we must shoot the gamepiece into the goal. The rest incorporates basic stuff, half of which I have completed. I learned some stuff in DLib, so it will be quite simple for me to write an application that has an amazing interface.

This gives me the question: If it is so easy to host a webpage without much processor usage footprint, would it be possible for us to give our alliance partners a web address where we can basically create a diagnostics system/chat system for each other to rapidly talk through?

Originally Posted by MatthewC529

Exactly. We know it isn't impossible. It's just the feasibility I personally feel is being downplayed. The feasibility of creating a robot that adapts to a given game in a meaningful way I think requires you to be able to look at it as more than a little OpenCV and clever data gathering. The details are important. Everything seems simple in the big picture until you break it down realtime. Though you are not wrong in saying AI/ML can be deceptively simple.

You think of this as a game in code because it is a game. It's why I use iterative. Reminiscent of a game loop. But unlike a game it's a lot more difficult to handle AI and ML in reality, let alone competition.

Sorry for spelling errors. Mobile is not friendly to my hands.

I really do not think that I have explained my goals to well to everyone. There are three things that would be nice to implement:
-Automatic gamepiece collection
-Automatic gamepiece scoring
-Automatic driving algorithm so the robot automatically gets to a location requested by the driver

Those are all the things required to make the robot self-capable of possibly winning a match. I guess the gamepiece collection algorithm can get a bit weird because you would need to make sure that you do not hit a ball of the other team, and so you don't attempt to collect a ball present within another robot.

However, proper camera calibration will mean that the height to y axis in the camera , according to the ground. I believe that the relation would be linear. This would allow the program to tell whether the gamepiece is available for pickup!

Originally Posted by StevenB

So, I'm a graduate student in electrical engineering, and have taken classes in machine learning, artificial intelligence, image processing, and computer vision. I've spent four summers in internships doing image processing with MATLAB and OpenCV.


Ok, I hesitate to call myself an expert, but I do know what I'm talking about.



I love MATLAB, and yes, it does let you do some pretty neat things with very little code. You can develop and prototype algorithms very quickly, and there seems to be a function for everything you want to do. I can't speak quite as highly of OpenCV, but it too allows you to quickly piece together a lot of computer vision building blocks to build new things. There are dozens of other great libraries out there too - stuff like ROS, VLFeat, PyML, and more.

But even with such great tools, I think you are radically underestimating the problem. Seriously, if you create a robot that can play Aerial Assist autonomously the way you're describing, you will have done enough innovative work to publish several papers and get a PhD.

I'm not saying this to discourage you. You've got a lot of passion and excitement, and a lot of great ideas, and I want to see something cool and innovative come out of it. You can do great work, and I look forward to seeing it. But I encourage you to focus your effort on a small piece of the problem. It may feel like a teeny, tiny, insignificant part. But unless you start with something small, it's hard to finish. I'm a dreamer by nature - like you, I have big ideas and grand plans, and I like to start new things. But too many of my projects have died shortly after I wrote down the grand vision, because I took on too much at once.

So, think about a small part of the problem, but also something you can get excited about.

• Program your robot to catch a ball thrown to it by a human. Use an onboard camera (looks like you've already got one on your bot) to track the trajectory of the ball, and have the robot drive into position to grab it.
• Place a series of traffic cones (or similar large bright objects) on your playing field. Make the robot chase down the ball and pick it up without knocking over any cones.
• Program your robot to automatically score the ball after picking it up a random distance from the goal, while avoiding traffic cones.
• Build the multi-camera system you described, and program the robot to play defense. A robot with red bumpers is trying to pick up a red ball. Your job is to stay between the red robot and the red ball.

These are all things that are doable in a summer, but to my knowledge, no team has done these. Pick one of these, or something similar, break it down into tiny tasks, and see what you can do. Ask questions, read papers, and write some code. With enough dedication, you'll come up with something really great, and I look forward to seeing it!


Someone should go tell the RoboCup teams. They've been working on this for 15+ years.

Well, there are a few goals that I have in mind. This isn't to build a robot program that will convert the crappiest robot build into a champion bot, but instead to build a prototype that performs a bit better than a novice driver who knows little about the robot.
Driving through traffic cones is quite the opposite I am trying to perform. Catching the ball would be a challenge, but I know that it has been performed by quite a few teams. Catching a ball without knocking over cones is also past my goals. I am not trying to build something that is so accurate that it will never hit anything. I know that on the field, if the robot just touches, or maybe takes a small hit, nothing bad will happen. I just want to make sure that the robot won't generate a path through the wall! The main thing that will help is that the path will be regenerated constantly in a seperate thread, so if one error happens, the robot will be out of control for less than an eighth of a second! I want to build a multicam system, however, the processing power, resources, etc. will be beyond the reach for my team, so I want to stick with two high-FOV cameras and set up stereo vision!

[JamesTerm]

Originally Posted by MatthewC529

You think of this as a game in code because it is a game. It's why I use iterative. Reminiscent of a game loop. But unlike a game it's a lot more difficult to handle AI and ML in reality, let alone competition.

Why would it be difficult to handle AI? I guess AI should have more clarity as it's a real generic term. When I say AI especially in terms of gaming I'm really saying what is written in this book. I loved this book as our code uses a very similar model of goal driven classes... the context of AI written here is really a way to manage events in real-time.

Given this context... the AI foundation for our game code we wrote , and the robot code are 100% identical.

I think your context of AI includes the details of the goals themselves... assuming this to be true, it should be abstracted away to not be a part of the AI... so let's take 2011 logomotion as an example, and only just the autonomous part as an example to illustrate my point:

goals
drive forward 10 feet
raise arm to 9 foot mark
drive forward one more foot
open raptor claw
lower arm 6 inches
drive backwards a few feet

In this example these are high level goals that I could have used in the game, the details of how to do the goals would have been different, and FWIW... driving forward *straight* is not as easy as it sounds.

[JamesTerm]

Originally Posted by yash101

Well, y'all are the experts, so I'd have to trust you on that. However, I have come up with quite a tangible and possible plan. The entire program will basically be split up into parts. Also, it will only have basic offense features.

I'll throw some things out there for you... sayings that I've found to be true:

1. People need to follow through their plans... just like a tennis player needs to follow through when he hits the ball on the serve. So follow your passions and see them through to the end. I hope you get some new innovation and learning experience and share with us the outcome!

2. Top down design, bottom up implementation. You now have the plan... break it down to small goals and start to form some implementation strategies. It's time to get down from the dream cloud to face reality!

3. Don't waste your time shuffling/organizing chairs on the titanic. As we get older we realize how precious time is and how little of it we really have. As a team player you have a responsibility to your team as well as following your passion... try to find a happy medium in there. Is your time invested in this taking away what you can contribute to your team? What is the biggest fire or issue with the team that needs help. What can you do this summer to be ready for next year... to overcome known mistakes learned from this season etc. My biggest regret in life is not being a better team player when I was younger.

The sayings in 2 and 3 come from my boss/mentor and I've adopted them as my own, and I'm passing them down to you. Good luck! Whatever the result... it will be a great learning exercise!

[MatthewC529]

Originally Posted by JamesTerm

Why would it be difficult to handle AI?

I think your context of AI includes the details of the goals themselves... assuming this to be true, it should be abstracted away to not be a part of the AI...

You are right in stating AI is a very generic term, you can throw around words like A* Pathfinding, State Machine and Goal Oriented Behavior but those can become literally just terms to throw around since a specific task usually requires a more specifically designed AI to be done effectively and, more importantly in games, efficiently.

And no that is not the context I am looking at it from. In reality if you write an overly specific AI focused on conducting only a few tasks than you end up writing a lot more unmanageable code (Learned the hard way, it was an invaluable learning experience), possibly resulting in several anti-patterns, when in reality an abstract approach goes a LONG way. Though you should know from writing that game that you cant just ignore the deeper details. If you dont account for the details you can end up with an exploitable and appearingly buggy AI.

Your example though is great, that is ideal, but as you said it's difficult to even account for the driving "straight" part, and that is what I am getting at. You can say a bit of "MATLAB and OpenCV" but in reality to have a truly effective AI you will need more. Having AI to achieve goals can be easy in a digital game, creating an AI to adapt to the player and other AI is more difficult and having an AI to adapt to the player and other AI with uncertainties in what may occur (particularly speaking about reality) then it gets more difficult. Again I love this idea and I love the plan but its also being over simplified in my opinion.