Python dictionary to handle large data -
i know how efficient python dictionary in handling big data. lets suppose have 2 12 gb files,can create 2 dictionaries each file 90% of info unique in each file.and next need these dictionaries comparing info based on keys.i tried on 16gb machine consuming memory , skipped executing script .or alternative dictionaries in python ? below sample code
for line in fileinput.input(src_one,0,"",0,'r',false): line = line.strip() tmparr=line.split('|') key=tmparr[2],tmparr[3],tmparr[11],tmparr[12],tmparr[13],tmparr[14],tmparr[15] dictone[key]=line line in fileinput.input(src_two,0,"",0,'r',false): line = line.strip() tmparr=line.split('|') key=tmparr[2],tmparr[3],tmparr[11],tmparr[12],tmparr[13],tmparr[14],tmparr[15] dicttwo[key]=line
thanks.
python dictionaries not designed out of memory data.
however there standard libraries this, namely shelve
module.
python
No comments:
Post a Comment