Check out our Video of HHH Match 5 Augmented with DataZ for FRC.
So we saw people posting in the thread about regional webcasts and thought it would be fun to release some of the work we have been doing! The DataZ Map View at the bottom could also be placed over a top down stream with team number tags rather than tails to solve the problem of not seeing bumper numbers. We have the technology… 
All the new stuff is awesome! I hope to see this technology integrated into events in the near future.
Have you considered overlaying the location data on the video? Should be a fairly simple transformation to convert X/Y into a 2d plane aligned with the field?
I’d have to look up the exact matrix to multiply it by because my graphics stuff is a bit rusty but maybe someone else knows?
We have been investigating that, and a few others have asked as well. It is JUST math after all If someone wants to work with us to speed that up let us know!
I actually suggested something similar to TBA a few months back, it could make watching livestreams much easier if you could easily identify teams from above which could be done with camera tracking and “name cards” that track with the robots.
Just throw more processing power at it, that usually works, right? :rolleyes:
Would the issue be more refresh rate (number of frames processed in a given time) or latency (delay between frame display and processing) related? If it’s just latency, simply delaying the video output so it matches the overlay should be adequate to compensate and easy enough to do with existing streaming software (and livestream viewers wouldn’t notice). If it’s refresh rate I can’t really help, though depending on how bad it is right now it might be acceptable to have short update gaps in data (so instead of smoothly following a robot it would jump every second or two). Just a thought.
I watched the video and started to wish John Madden could come out of retirement and so commentary on events. He was the best with the telestrator, this would be in his wheelhouse.
Nice job on the videos!
Doing it live might be hard. But doing it on the video afterwards is just a simple function of computing the right perspective matrices and multiplying your position by that. The issue is your perspective matrix is based on the camera location.
Would the slightly slanted views from PNW dcmp this year work, or are we needing those straight top down views from St. Louis?
So, that’s the thing, you can use whatever view you have as long as you can properly convert between the two coordinate systems (which is all the transform matrix does).
wouldnt it be a pain for the viewer though if the camera was fairly level to the field? Sure, it can be done, but will it be viewable?
Yes and no. You could, with a 3d model of the field structures compute occlusion of models and be sure to highlight them differently so you would give the audience watching the stream X-ray vision in a sense.
Basically, this enables a sort of augmented reality view. To start with showing just the robot positions is low hanging fruit. But adding in things that make the game easier to follow or highlight useful things on the field for the audience? It’s the same concept as the first down line in football. That line is drawn on by computing the location of the camera relative to the field and drawing a line.
Just to expand on this, the first down line works despite having moving cameras because the position of the cameras are calibrated beforehand and the pan/tilt of the cameras is measured on the fly.
For our application in FRC, as long as a camera doesn’t move (such as a full field camera), one-time calibration is needed. This calibration can be as simple as picking out a few “known” points on the field (such as the 8 corners of the red and blue driver station glass walls).
We can even predict field occlusion (such as by the airships in 2017) to show different “augmented reality” overlays whether the robot is behind or in front of these field occlusions.
I would think the type of lens would affect it too, since a fisheye lens would act very differently from a normal lens and affect the result.
That said, if you’re using it purely as a tracking overlay for visual labels, then as long as you have an overhead camera, even at a small angle (say, 0-20 degrees), the output would probably still be accurate enough for those uses without any adjustment for angle (we don’t need super-high precision as long the result close enough).
Doing it live might be hard, but I don’t think anyone would be complaining about a few second delay to streams.
You could probably get away with a few degrees off vertical. But it would drive people batty over time. This is one of those things if you miss by and inch you’ve missed by a mile. At the point where a true overhead camera isn’t available a virtual field with robot markets would likely be a better substitute.
Eh, matrix operations are quick tbh. Heck, if you want to see an example of the transform being run hop into chrome and open the inspector, select an element and add the following css to it
transform: rotateX(70deg) rotateZ(30deg)
It will very quickly snap to a different orientation. I actually looked into using this to rotate the field in my viewer but ultimately decided that the top down view was easier.
Currently overlaying DataZ data on a perfect top down view of the field is straight forward and could be done rather quickly with our calculations needing a bit over .5 second delay which is probably acceptable for an online “live” stream. As Andrew said, adjusting for oblique field angles can be done, as could modifying the view to account for field obstructions with fixed location and orientation cameras. This could probably be kept on a similar delay. As for Fish Eye, provided the distortion is consistent it could be accounted for. This would of course require us to be set up at the event to collect the data. SBPLI has expressed their interest to run DataZ for FRC at their 2018 events pending FIRST approval (already contacted).
I’ve actually been working on something relevant in openCV recently, localizing a camera using pictures of a square grid, so I figured I might chime in.
Implementing this shouldn’t be too hard, even while trying to account for camera distortion. The OpenCV calibratecamera function takes in a list of 3d world coordinates and their corresponding image coordinates, and returns a camera matrix. I think it also returns a translation and rotation vector relative to the object, although I haven’t actually used that part (I was using SolvePNP on the same page for that).
With camera matrix, translation, and rotation determined, conversion from 3d to image coordinates should be a simple multiplication. Only worry is the potentially small number of data points for determining camera matrix. And the headache of debugging.
I’m extremely interested in this! Being one of the A/V guys at the PNW, I love trying new things! Would this be easier if there were some sort of tracking markers? For instance, could you find IR LEDs (powered off of an Arduino or something easy) to make the math easier? How would you overlay the name cards so as not to obstruct the robot too much? I would be really interested in talking with the team that’s developing this to see if it’s something we at the PNW could implement (maybe for our DCMP or something!)
Ok, so I have to admit, when I first read through this thread I was not aware of what DataZ was or how it worked (I had been assuming some kind of vision tracking initially). That said after looking up and reading through the other threads on this topic (For those of you who haven’t seen the other DataZ threads I HIGHLY recommend looking them up), I have to say I’m pretty excited about some of the potential possibilities…
For Drive Teams:
For Programmers:
For Spectators/Livestream viewers:
Granted I assume this is all still very WIP, but it will definitely be interesting to see how it turns out. Have you guys gotten interest from FIRST about this sort of thing? As a member of my local district event planning committee I can say we would definitely be interested in accommodating any sort of pilot you guys might be planning (we also do an off-season event every year in the fall).
One concern I do have about a system like this in the long-term is operational logistics… Ideally (particularly for smaller events) this system would require minimal additional effort to set up at an event, and better yet if it functioned without any user intervention (since many events are strapped for volunteers as it is). We ran into a similar issue when we were developing an AV system for our event that was intended to also be loaned to other events. In our haste to make the system flexible, high quality, and low cost, we made the mistake of making it so elaborate to setup and run that we basically have to send someone with it to every event and then recruit an additional 2-4 volunteers to help operate it at the event (needless to say we’ve been trying to streamline the system over the years to help alleviate this).
In any case, still very interested to see what comes of all this, I think a lot of us have been hoping for something like this to be available for a long time. Keep us up to date (and keep posting pictures)! 
One of the biggest potential values, I think, is to automate scouting/statistics. If this was a thing in 2017, we could have accurate climbing rates without having to watch any matches – just compare the field record of which touchpads scored with where robots ended the match. Additionally, it could provide some more advanced stats like average time spent at loading station or average cycle time. If FIRST continues to provide more information through their API, including when certain things happen, this will help to attribute more actions to specific teams.
I’m not as bullish using this technology to help live streams or re-watching matches. If there is decent match video, I’m going to watch that over just colored lines. Overlaying the team number onto the video sounds interesting, but I think it’s only really helpful at the start of the match, similar to how the NFL utilizes it.