numpy - getting Killed message in Python -- Is memory the issue? -

numpy - getting Killed message in Python -- Is memory the issue? -

i have list .append() in for-loop, length of list around 180,000. each item in list numpy array of 7680 float32 values.

then convert list numpy array, i.e. expect array of shape ( 180000, 7680 ):

d = numpy.asarray( dlist, dtype = 'float32' )

that caused script crash message killed.

is memory problem? assuming float32 takes 4 bytes, 180000x7680x4bytes = 5.5 gb.

i using 64 bit ubuntu, 12 gb ram.

yes, memory problem

your estimate needs take business relationship memory-allocation done list-representation of 180000 x 7680 x float32, without other details on dynamic memory-releases / garbage-collections, numpy.asarray() method needs bit more space of 1800000 x 7680 x numpy.float32 bytes.

if seek test less 3rd length of list, may inspect resulting effective-overhead of numpy.array data-representation, have exact info memory-feasible design

memory-profiling may help point out bottleneck , understand code requirements, may help save half of allocation space needed data, compared original mode of data-flow , operations: (fig.: courtesy scikit-learn testing numpy-based or blas-direct calling method impact on memory-allocation envelopes )

python numpy