loading a COO file into Python

[virtuald]

Apparently pandas has something for this that outperforms numpy.loadtxt?

http://wesmckinney.com/blog/a-new-hi...ne-for-pandas/

And here:

http://akuederle.com/stop-using-numpy-loadtxt

[Ether]

Thanks Dustin.

I'll try that.

[Ether]

Originally Posted by virtuald

Apparently pandas has something for this that outperforms numpy.loadtxt?

WOW. Pandas is FAST.

Just ran it on a slower machine and it loaded Aijv.dat in less than one second.

You are my Python guru Dustin

[virtuald]

Originally Posted by Ether

WOW. Pandas is FAST.

Just ran it on a slower machine and it loaded Aijv.dat in less than one second.

You are my Python guru Dustin

Glad to help.

My guess is that because data isn't already a numpy array, it has to do a lot of work to convert it to one first, and then do the transpose operation. If I'm correct, a second transpose on the result of the first transpose would be very fast, similar to the operation on Ajiv.

If I understand what transpose does, it most likely is just moving the pointers to the various axes around -- thus why it's so fast. My understanding of how numpy arrays work is that they try really hard to not actually move data around, but many common operations can be done by just creating/moving various pointers around.