DataZ for FRC: Mo DataZ Mo Problems?

Hi again everyone! Yes, we know, we said Cyber Monday and it is now Tradesman Tuesday. As the title suggests, while sifting through these data points and developing the applications to parse and display the results we hit a few snags. These issues were not entirely unexpected but sadly delayed this release.

We are now on imgur!

We have uploaded a gallery for your viewing pleasure that includes Heat Maps for all Qual matches at the Half Hollow Hills Invitational.

We have updated our YouTube channel with 7 more videos!

We have a nice collection of data playback for some elimination matches at about 4x speed. We will continue adding video playbacks if the community finds them interesting. To start with, we tried picking matches to show every robot at the event at least once.

Teams at the event: If we missed you, post on here and we will find and upload one of yours!

**imgur: **DataZ4FRC

**YouTube: **DataZ 4FRC

As always, questions, comments, and ideas are always welcome!

PS: Yes Andrew Schreiber we are still working out the best way to distribute the raw data (Team, Center X, Center Y, Orientation, Time, etc…) :wink:

Are the points on the heat maps separated by an even time interval? Can we use the spacing to calculate speed?

Hey, I’m not picky. You should see some of the utter garbage I’ve gotten from government data. It’s like they went out of their way to make it as unusable as possible. As long as you don’t make me do a join on a normalized street address to get useable data I’ll call it a win.

First of all, I don’t think I’ve posted on any of these threads yet, so I’ll say this technology is awesome. I’d love to see it at every FRC event, perhaps even integrated into the FMS someday (hey I can dream :rolleyes:)

Second, I’m assuming you’re open to feedback. I am having a hard time distinguishing between the different shades of red and blue, especially when they’re not right next to each other. I like the idea of the red alliances being shades of red and blue being blue, but I’d rather have more different colors that are easily distinguishable. It’s not as bad in the video playback, but it’s still hard to tell which robot is which. Also, no matter what, a key showing which team is which color would be very helpful.

This data looks awesome! I’m glad we participated.

Yes! and YES! Those data points are at even 100 ms intervals synchronized across all 6 robots. That is exactly how we were calculating the velocities in the videos.

We think we share the same dream. We are working hard to make that dream come true! Yes we love feedback. As for the colors for the heat maps and match playback, we went back and forth on color combinations and so when we release the viewer we plan on letting the user select colors. A key! D’OH >.<

Thanks again for your support and participation!

Andrew, Pm sent.

This data would be insanely helpful for strategy and scouting on just about every team. Thank you guys, and I think this has the potential to be a game changer for FRC.

We are making the DataZ data for the matches shown on our YouTube channel available at the following link:

http://goo.gl/nhua8k

This links to a folder on our Google Drive where we have matches stored.

Each folder contains 6 CSV files. The first line of each is Team Number and Match Number
The following lines are the x and y coordinates on the field in feet.
(0,0) is the corner closest to the Red Boiler.
X is from 0-54 feet, y is from 0-27 feet
the third column is time

We are interested in seeing what the community can do with DataZ and so the rest of the match data will be made available mid December!

Just for giggles - here’s some gifs from a VERY rudimentary visualizer (they are actually a nice buttery smooth animation live but gifs…)

358’s movement in match number 12

All teams in match 201

And just for giggles - 4 matches worth of data all at once

I’ll work on cleaning it up tomorrow and adding error handling code so it checks that what you dropped in is a csv file.

Now with video… and color https://youtu.be/ABw6mZtSmtw

This type of data is very exciting. My team has been doing location-based scouting since 2013 which requires students with tablets tapping locations on a playing field and hitting buttons to say what happened. This would alleviate so much work from the scouter standpoint. I really hope this data will be available in the coming FRC seasons.

This is very cool. Is there a way to overlay the game elements? That would let teams see what they were doing. I’m always on my drive team to “drive with a purpose” Dataz gives a way to see how well they getting to feeders and goals.

Thanks

Sorry, just went back to watch the video. That’s what I was looking for!!

Made some css tweaks to make it less ugly, apparently broke something in Safari so for now - Chrome only.

https://schreiaj.github.io/dataz-viz/

I didn’t provide any data since I never really got an answer from Zebra about the license on the data files and didn’t want to redistribute anything I didn’t have permission to.

Just drag and drop any number of the csvs provided by DataZ into it. Remove file by clicking on the team_no/match label.

May get cleaned up over the next couple days depending on work stuff.

We saw this and it looks pretty cool! We like the idea of a drag and drop for the CSV files. The DataZ files for the HHH invitational already released above are not licensed at all and so are free for anyone to use. The CSV formatting will likely change slightly in future versions as we continue to evaluate the system. We will also be releasing our sample viewer code, as is, under some Open Source licensing arrangement. Coinciding with the release of the rest of the DataZ for the HHH invitational we are also looking into hosting a Hackathon for teams interested in working with us on DataZ viewers and a CADathon for teams interested in creating extensible tag to robot connection attachments. More details on those will follow, if enough people seem interested in this.

Cool, if it changes I’ll push some updates.

Some requests, having different rows representing different things is a serious pain as most CSV parsers want to use the first row as either headers or data which isn’t valid in this case. Heck, it might even make more sense to replicate that data as columns itself. This would facilitate loading of the data into a database more effectively at the cost of data normalization and file size.

Normalizing the time, while easy enough to do manually, would make it far easier to approach the data. I’m cheating and just stepping through the observations so I can ignore the time stamps but using match time steps might make life easier for other folks.

I know Andrew already posted a pretty neat one but here was my take on the data viewer: https://danwaxman.github.io/DataZViewer/. It’s very similar to Andrew’s but has play/pause/speed control.

I had limited time this weekend so right now it’s just a bunch of really hacky JS but I’m planning multiple team and CSV support some time tonight.

Edit: So far I added an image of a field and multiple team with CSV Support.

The length of the CSV files comes out to about 1850 rows, which, if the data points are at 100ms intervals, comes out to 185 seconds. The robots seem to start moving immediately at the beginning, and the match is definitely not over at 150 seconds, so I’m not sure where this extra data comes from. Any ideas? This is a pretty big concern if we’re calculating velocities from this data.

Perhaps there’s data on the end of the files after the buzzer. Most of the I’ve looked at show the bots clustered around the ropes and some of them moving slightly (i.e. swaying)

I’m just theorizing here.

As this was a Beta without FMS integration, all data collection was started and stopped manually. This means some match data doesn’t start right away and some match data ends way after the match. If you could let us know what ones you found an issue with that would help us improve the system. This is why we released some of the data and is a part in our delay with releasing the rest.

So, I have a question that will help lend context to all this new data: what were the alliances and playoff results of HHH?

For the playoffs at this event a single match was played until the finals which became best of 3.

For the matches shared (Match, Blue1, Blue2, Blue3, Red1, Red2, Red3):

Quals:
12, 3624, 2869, 5659, 358, 1468, 5099
17, 329, 6423, 5736, 2869, 514, 263

Playoffs: 1,3,4,5
101, 810, 2872, 6706, 6746, 5016, 3950
103, 329, 5736, 353, 5099, 6423, 6422
104, 2875, 358, 1751, 527, 871, 4458
105, 870, 3171, 514, 810, 2872, 6706

Finals 1:
201, 527, 871, 4458, 810, 2872, 6706

No, we did not run 201 matches :wink:

The winning Alliance was the Blue Alliance, teams 527, 871 and 4458! Congratulations!
These particular matches were selected to allow visibility of the most amount of teams in the competition, and highlighting the playoffs and finals