Cool project idea: LV to play a game or program

Technical NI LabVIEW
1

I’ve been thinking about this project idea; what do you think and how could it be done?

You have a LabVIEW VI play a simple computer game, analyzing the simple situations and “clicking” in the right spots to play it. Specifically, I’ve thought of Facebook’s Bejeweled Blitz, because the computer could very quickly find rows to make. Maybe there’s something easier to implement out there?

Not only would this be impressive, but it would probably be a good learning opportunity for FRC teams in advanced programing (or just more advanced than they’ve done so far).

**
Applying this to Bejeweled Blitz: (gem game where you swap two adjacent gems on a grid to make 3 in a row or more)**.
LabVIEW could look at the pixels of the screen (or a snapshot of the screen) and analyze the center pixel of each gem position, finding the color of the gem. In this way it would what the grid situation looks like.

To make moves, click events could be generated on gems that needed to be moved. (I’m personally not very knowledgeable about using event structures and generating events, though)

To decide what to do, every gem could be analyzed in a couple of ways:
-See if possible rows are near: look at similar colored gems diagonal from this gem. If there are two diagonals in a similar direction (up, down, right, left) or another similar gem next to the diagonal, move this gem next to the diagonal to make a row.
-See if a row would be made if this gem was moved in X direction: theoretically move the gem, see if two similar gems are next to it in a row.

The sliding down of new gems after a move would have to be considered. They take time to actually get into position and you can’t test when gems are moving by. The simplest way would be to wait for a key press from the user signifying all gems have fallen and have stopped making any combos. You could continue analyzing gems that are not above the played row until after a short time limit. This wouldn’t take into account any combos, though.

So what do you think? I don’t have LabVIEW with me on a computer, so I can’t look at what VIs I’d use or go more in depth on how I’d do it. Are there any other strategies for doing what needs to be done?

2

I’m pretty sure events are notifications that something has happened on the UI; they don’t cause things to happen on the UI.
However, if you can find a way to move a mouse around with native LabVIEW, do tell us.

3

This made me think of the Mario AI Competition that had been held for the last couple of years. It’s in Java instead of LabVIEW, obviously Mario’s a bit more of a challenge than Bejeweled, and the deadline had passed, but maybe we could hold a CD-version of something like that.

Back in the day, I used to make calls to the Windows API in LabVIEW using the Library Call node, which would make it pretty easy to move the mouse around and click (assuming you’re using Windows). There’s also ways to capture screenshots, but then you’d have to do the requisite machine vision on the captured image (but it should be pretty easy with a couple of Threshold Color -> Particle Filter -> Extract Particles). Search for ‘windows api labview’ on Google and it should turn up some resources, but it would probably be helpful to have some knowledge of C/C++ (or ask a mentor :)).

Sounds like a fun project.

–Ryan

4

Now that you say that, I think you may be right. I’ve always thought VIs and event structure operations like “generate mouse click event” actually generated a mouse click, but maybe not. There’s got to be a way to do it though.

Thanks for the idea on manipulating the Windows API!

5