python - How to NOT load whole file into memory during upload -

python - How to NOT load whole file into memory during upload -

am using bottle create upload api. script below able upload file directory got 2 issues need address. 1 how can avoid loading whole file memory other how set maximum size upload file?

is possible continuously read file , dump has been read file till upload complete? upload.save(file_path, overwrite=false, chunk_size=1024) function seems load whole file memory. in tutorial, have pointed out using .read() is dangerous.

from bottle import bottle, request, run, response, route, default_app, static_file app = bottle() @route('/upload', method='post') def upload_file(): function_name = sys._getframe().f_code.co_name try: upload = request.files.get("upload_file") if not upload: homecoming "nothing upload" else: #get file_name , extension file_name, ext = os.path.splitext(upload.filename) if ext in ('.exe', '.msi', '.py'): homecoming "file extension not allowed." #determine folder save upload save_folder = "/tmp/{folder}".format(folder='external_files') if not os.path.exists(save_folder): os.makedirs(save_folder) #determine file_path file_path = "{path}/{time_now}_{file}".\ format(path=save_folder, file=upload.filename, timestamp=time_now) #save upload file in chunks upload.save(file_path, overwrite=false, chunk_size=1024) homecoming "file saved {0}{1} '{2}'.".format(file_name, ext, save_folder) except keyboardinterrupt: logger.info('%s: ' %(function_name), "someone pressed cnrl + c") except: logger.error('%s: ' %(function_name), exc_info=true) print("exception occurred111. location: %s" %(function_name)) finally: pass if __name__ == '__main__': run(host="localhost", port=8080, reloader=true, debug=true) else: application = default_app()

i tried doing file.write same case. file getting read memory , hangs machine.

file_to_write = open("%s" %(output_file_path), "wb") while true: datachunk = upload.file.read(1024) if not datachunk: break file_to_write.write(datachunk)

related this, i've seen property memfile_max several so posts claim 1 set maximum file upload size. i've tried setting seems not have effect files no matter size going through.

note want able receive office document plain extensions or zipped password.

using python3.4 , bottle 0.12.7

basically, want phone call upload.read(1024) in loop. (untested):

with open(file_path, 'wb') dest: chunk = upload.read(1024) while chunk: dest.write(chunk) chunk = upload.read(1024)

(do not phone call open on upload; it's open you.)

this answer includes more illustration sof how read big file without "slurping" it.

python bottle