Twitter decoding program

[Lalaland1125]

I have now added least squares solving in order to better find the "impact" each team had on the score. The results are now much more accurate, and better predict the results in the final match( the total of the average scores from each team member is around +- 5 from the total score for that team(excluding outliers like team 93)).

( I am only using data from the qualifying matches to predict the final rounds)

While of course hand recording the individual scores of each team would be more accurate, this should be a great help in determining which teams provide the most "positive" points to help in the finals.

[Ether]

Originally Posted by Lalaland1125

I have now added least squares solving in order to better find the "impact" each team had on the score. The results are now much more accurate...

Just a heads-up: the Twitter data has completeness issues:

http://www.chiefdelphi.com/forums/sh....php?p=1144595

http://www.chiefdelphi.com/forums/sh....php?p=1144727


Oh, and a couple questions: What linear algebra library are you using, and is there a reason you are using SVD?

[Lalaland1125]

The math library is Eigen.

The reason why I am using SVD is because that is how Eigen's tutorials describe how to perform a least squares operation(http://eigen.tuxfamily.org/api/Tutor...Leastsqua res)

I don't think missing scores is going to be that bad. As long as most of the scores are posted, there should be enough data to get a reasonably accurate result. If anything, the main problem with my model is that it is very limited, not counting defense, autonomous, etc

[Ether]

Originally Posted by Lalaland1125

The reason why I am using SVD is because that is how Eigen's tutorials describe how to perform a least squares operation(http://eigen.tuxfamily.org/api/Tutor...Leastsqua res)

SVD is just one (of many) ways to compute least squares. The choice of the "best" method to use (like choosing the right tool for a job) depends on the problem domain.

For this application, LDLT would be far faster* and plenty accurate.

I don't think missing scores is going to be that bad. As long as most of the scores are posted, there should be enough data to get a reasonably accurate result.

Be aware: There are already two events (Oregon and Traverse City) for which data is missing for the entire event. No guarantee that won't happen for more events as the season rolls on. Not counting these two events, over 12% of the data is missing for the other events.



* For computing least squares for single events, the matrix is small enough that the time difference is probably not even noticeable. But if you ever intend to expand the functionality to compute least squares for a matrix containing all the data from an entire year's worth of events, I believe there would be a very noticeable difference in speed. If you have the time and are so inclined, it would be interesting if you would try SVD with 2011's data and see what the computation time is. For reference, LDLT takes 12 seconds on my 8-year-old PC to do least squares on a matrix populated with all the qual data from all the events in 2011

[Ether]

Originally Posted by Ether

Be aware: There are already two events (Oregon and Traverse City) for which data is missing for the entire event. No guarantee that won't happen for more events as the season rolls on. Not counting these two events, over 12% of the data is missing for the other events.

Twitter data is missing entirely for the San Diego, Oregon, Traverse City, Sacramento, and Utah events.

For this application, LDLT would be far faster* and plenty accurate.

The alliance selection algorithm in the qualification match scheduling software used for FRC events pretty much guarantees that the design matrix A (i.e. Ax≈b) will be full rank and well-conditioned. This means that forming the normal equations (Px=A^TAx=A^Tb=S) and solving with Cholesky decomposition (LL^T or LDL^T) will give excellent numerical stability and accuracy and be far faster than other methods (and require much less memory). Furthermore, the normal equations can be formed directly in one pass (without the need to form the design matrix and multiply it by its transpose) when the raw data is read and parsed.

[kmehta]

Originally Posted by Ether

Twitter data is missing entirely for the San Diego, Oregon, Traverse City, Sacramento, and Utah events.

Regarding Sacramento, I talked with the field crew/FTA and their Twitter posts were apparently being blocked by the firewall. If the same happens at SVR, we will have a team member manually copy the data down so it won't be lost.

[Ether]

Originally Posted by kmehta

Regarding Sacramento, I talked with the field crew/FTA and their Twitter posts were apparently being blocked by the firewall. If the same happens at SVR, we will have a team member manually copy the data down so it won't be lost.

It is admirable and I applaud your team for being willing to do that.

But in all seriousness, why should that even be necessary?

This data has significant statistical value and historical interest. The data is already in electronic form. Does anyone know: Is there a compelling reason why the data is being discarded instead of being saved at the point of origin?