R Package for Downloading FIRST API Data

[Ether]

Originally Posted by sirwin

my solution is posted at https://irwinsnet.github.io/opr.html.

My method requires 13 seconds on a Surface 4 with an i5 process and 8 Gb of memory.

Reps to you for all the fine work

Reps to the first person who can cut that time in half.

[Foster]

Originally Posted by Ether

Reps to you for all the fine work

Reps to the first person who can cut that time in half.

Cut 13 seconds in half? I'm excited that sirwin got it to work at all on that data set! At 13 seconds, the WOPR rankings can be recalculated between matches before the first verse of "Cotton eye Joe" gets played.

Good job sirwin on the code, and an even better write up of how you did it.

[sirwin]

With respect to speeding up the calculation, I had a thought.

It takes my algorithm 9 seconds to produce the 2696 x 2696 A matrix -- but updating the A matrix with new match data should be nearly instantaneous. For every new match, we just have to add 1 to 18 different elements in the A matrix (all other A matrix elements will remain unchanged). Each element of the a matrix is directly accessible by team numbers. So it takes 13 seconds to calculate world OPR from scratch, but as long as we retain the A matrix, it should only take about 5 seconds to update world OPR with new match data. That's 2 seconds for updating the A and B matrices and 3 seconds for solving for OPR.

Of course, this is assuming that R's underlying C code doesn't do anything stupid, like extracting an array element by going through every array element until it gets to the right one. Also, the A matrix would need to be maintained in memory -- disk access would slow things down.

[Ether]

Originally Posted by sirwin

It takes my algorithm 9 seconds to produce the 2696 x 2696 A matrix

I like to call that symmetric positive definite 2696x2696 normal equations matrix the N matrix... to distinguish it from the 17842x2696 dichotomous design matrix A.

Ax = b .. overdetermined system of linear equations (has no exact solution)

N = A'*A
d = A'*b

Nx = d .. linear system of normal equations (whose solution is the least squares solution for Ax=b)

[Ether]

Originally Posted by Foster

Cut 13 seconds in half?

OK, I was being generous. Cut it by a factor of 10. Seriously.

[Ether]

Originally Posted by Ether

Cut it by a factor of 10. Seriously.

If there is any interest in learning how to do this within the R community please let me know and we can work it out together.

[Joey1939]

I took the world OPR data and made a graph.

EDIT: I made a better graph.

[Ether]

Originally Posted by Ether

If there is any interest in learning how to do this within the R community please let me know and we can work it out together.

https://www.chiefdelphi.com/forums/s...09#post1614009

[sirwin]

I created version 2.0.0 of the firstapiR package for downloading and manipulating FIRST API scouting data using the R language.

Here's a link:

http://irwinsnet.github.io

Once I started using the package to do OPR calculations, I realized it needed some improvements, specifically in how the data frames are shaped.

Stacy Irwin

[Ether]

@ Stacy: thanks for posting this. I just send you a PM.

[Skyehawk]

Originally Posted by sirwin

My method requires 13 seconds on a Surface 4 with an i5 process and 8 Gb of memory. It actually takes more time to create the A matrix (9 seconds) than it does to solve for the OPRs once the A and B matrices are prepared (3 seconds). The remaining second is required for reading the data and preparing the B matrix.
Stacy

I am aware I am beating a dead horse here , but using unmodified Sirwin's code I am getting ~12 seconds with an i7-6560U (2.2GHz) and 8gb of RAM, memory usage is low so we can eliminate that as a factor. The bottleneck is (unsurprisingly) CPU clock cycle speed.

I'm using firstapiR v2.0.0

Skye Leake

[Ether]

Originally Posted by Skyehawk

I am getting ~12 seconds with an i7-6560U (2.2GHz) and 8gb of RAM, memory usage is low so we can eliminate that as a factor. The bottleneck is (unsurprisingly) CPU clock cycle speed.

I am getting ~0.93 seconds on a 10-year-old Pentium D desktop running 32-bit XP with 1G RAM.

[Skyehawk]

I obviously am doing something not quite right...

[Ether]

Originally Posted by Skyehawk

I obviously am doing something not quite right...

Your results look about right for R code that uses dense matrix technology.

https://www.chiefdelphi.com/forums/s....php?p=1612297

https://www.chiefdelphi.com/forums/s....php?p=1612581

https://www.chiefdelphi.com/forums/s....php?p=1613118

https://www.chiefdelphi.com/forums/s....php?p=1614125

[Conor Ryan]

In case anybody wants to learn R, here is a great way to get introduced http://swirlstats.com