Thursday 15 April 2010

Python dictionary to handle large data -



Python dictionary to handle large data -

i know how efficient python dictionary in handling big data. lets suppose have 2 12 gb files,can create 2 dictionaries each file 90% of info unique in each file.and next need these dictionaries comparing info based on keys.i tried on 16gb machine consuming memory , skipped executing script .or alternative dictionaries in python ? below sample code

for line in fileinput.input(src_one,0,"",0,'r',false): line = line.strip() tmparr=line.split('|') key=tmparr[2],tmparr[3],tmparr[11],tmparr[12],tmparr[13],tmparr[14],tmparr[15] dictone[key]=line line in fileinput.input(src_two,0,"",0,'r',false): line = line.strip() tmparr=line.split('|') key=tmparr[2],tmparr[3],tmparr[11],tmparr[12],tmparr[13],tmparr[14],tmparr[15] dicttwo[key]=line

thanks.

python dictionaries not designed out of memory data.

however there standard libraries this, namely shelve module.

python

No comments:

Post a Comment