Is it possible (or legal) to use a windows application in addition to the Dashboard, that communicates with the cRIO?
My intention is writing a program in windows that will take the camera images from the cRIO, apply some image processing code, and send some data back to the cRIO.
Yes, this is possible and, assuming all other rules are followed, legal. If you look at the SmartDashboard project on FIRST Forge (http://firstforge.wpi.edu/sf/projects/smartdashboard), you will see that there is the ability to customize the SmartDashboard to perform image processing. The full documentation is still in process, but you should be able to skim through the code and see how it is working.
R80 just says that the software you write may NOT control the operating mode or state. If can do a whole heck of a lot of other stuff, like all of the dashboard functionality or image processing. R58 says “teams are expected to customize their Dashboard code.”
Adding digital control over aiming your shooting mechanism after target has been acquired sounds like a good idea.