loading a COO file into Python

[vScourge]

This only took 3.4 seconds to run on my Win 8.1 system.

import os
import time

time_start = time.time( )

data = [ ]

for line in open( r'D:\temp\Aijv.dat', 'r' ):
	data.append( [ int( x ) for x in line.rstrip( ).split( ' ' ) ] )

print( '{0:.2f} secs'.format( time.time( ) - time_start ) )

[Ether]

Originally Posted by vScourge

This only took 3.4 seconds to run on my Win 8.1 system.

Thanks. That's quite an improvement.

It took ~5.3 seconds on my 10-year-old PentiumD Win32 machine:

>>> start = time.time( )
>>> data = [ ]
>>> for line in open( r'k:/data/Aijv big data.dat', 'r' ):
...     data.append( [ int( x ) for x in line.rstrip( ).split( ' ' ) ] )
...
>>> time.time()-start
5.296999931335449

>>> start=time.time()
>>> Aijv = np.loadtxt('k:/data/Aijv big data.dat', 'int');
>>> time.time()-start
18.858999967575073

Also:

>>> start=time.time()
>>> Ajiv = np.transpose(Aijv)
>>> time.time()-start
0.0

>>> start=time.time()
>>> datajiv = np.transpose(data)
>>> time.time()-start
3.937999963760376

Why does it take so much longer to transpose data than Aijv ?


BTW, the benchmark time on my machine appears to be about 0.5 seconds. That's the total time it took a compiled Win32 app to read Aijv.dat into a transposed array.