2015 OPR After Week Three Events

[Ed Law]

Thank you for being patient. The World OPR now has the components published. The memory problem I had was due to the A(T) * A operation. With 40-60 teams, that is not a big deal. But when I calculate the World Component OPR, after 3 weeks, we have about 1800 teams. That is a 1800 X 1800 matrix and a total of over 6 billion multiplications and 6 billion additions.

I finally have to swap the rows and columns of the matrix to reduce the amount of memory paging in and out. It still took over 10 minutes on my i5 computer with 4 GB of memory. It is done. I am not looking forward to when there are 2900 teams after 6 weeks of competition. It will take over 40 minutes just to calculate that A(T) * A.

I need a faster computer to do this! May be I will borrow our team's CAD workstation to run this. It has an i7 and tons of memory.

[Ether]

Originally Posted by Ed Law

I need a faster computer to do this!

Ed, see the email I just sent you. You can generate the [N] matrix directly (with surrogates omitted) from the raw match results data, bypassing the [A']*[A] operation. [N] will still be symmetric positive definite if you do it right. Let me know if you need more info.

FWIW:

Tools like Octave support sparse matrices. Perhaps the most recent versions of VBA also do?

Using sparse matrix technology, Octave takes only 16 milliseconds to compute [N]=[A']*[A], and another 16 milliseconds to compute [x]=[N]\[d], for 1812 teams and 7292 alliance scores (3646 matches).



PS: This might be a quick fix: Does your version of VBA support integer multiplication of integer matrices? Both [A] and [N] are integer. Set the integer size to the native word size of your machine (probably 64-bit).

PPS: I attached the [A] and [N] matrices if anyone is curious.

[Citrus Dad]

Originally Posted by Ed Law

Thank you for being patient. The World OPR now has the components published. The memory problem I had was due to the A(T) * A operation. With 40-60 teams, that is not a big deal. But when I calculate the World Component OPR, after 3 weeks, we have about 1800 teams. That is a 1800 X 1800 matrix and a total of over 6 billion multiplications and 6 billion additions.

I finally have to swap the rows and columns of the matrix to reduce the amount of memory paging in and out. It still took over 10 minutes on my i5 computer with 4 GB of memory. It is done. I am not looking forward to when there are 2900 teams after 6 weeks of competition. It will take over 40 minutes just to calculate that A(T) * A.

I need a faster computer to do this! May be I will borrow our team's CAD workstation to run this. It has an i7 and tons of memory.

It may be that you have to move to a statistical / econometric package like STATA which can handle larger data sets? Or have you tried R?

[Ether]

Originally Posted by Citrus Dad

Or have you tried R?

Are you an R power user?

Perhaps you could answer this question for me.

Note: The question is specifically how to do the LAD computation using R. I have already successfully used AMPL & GAMS (MINOS, Gurobi, MOSEK, OOQP), Python (CVXOPT), and Octave (GLPK); I'd like to compare R's performance.

[Citrus Dad]

Originally Posted by Ether

Are you an R power user?

Perhaps you could answer this question for me.

Note: The question is specifically how to do the LAD computation using R. I have already successfully used AMPL & GAMS (MINOS, Gurobi, MOSEK, OOQP), Python (CVXOPT), and Octave (GLPK); I'd like to compare R's performance.

Unfortunately, my partner is the R power user. He extols its virtues and I'm passing them along. I have used STATA and I think it could easily handle the matrix and exceptions that Ed is facing.

[Ether]

Originally Posted by Citrus Dad

I have used STATA and I think it could easily handle the matrix and exceptions that Ed is facing.

Is there a STATA plugin for Excel? If not, I am almost certain Ed has no interest in starting all over with a new tool. What he's seeking is a way to make the memory requirements and computation time reasonable with what he already has.

That is why I did not suggest that he switch to Octave, which is free, and easy to use, and very fast (see my previous post).

I made three suggestions:

1) Find out if VBA now supports sparse matrices. If so, that is arguably the best solution.

2) Make the A an N matrices integers and do integer matrix multiply (if VBA supports it) to create N=A'*A . If that works, it would be a fairly simple fix.

3) Generate the N matrix directly from the raw match results data, without incrementing the appropriate 4 matrix elements (2 for each alliance partner) for each surrogate team. This involves more coding, but it would avoid the A'*A computation that is presently taking so much time in the spreadsheet.