I have the following code to convert a data(row data from postgress) to json. Usually len(data) = 100 000
def convert_to_json(self, data):
s3 = self.session.client('s3')
infos = {
'videos':[],
'total_count': len(data)
}
for row in data:
video_id = row[0]
url = s3.generate_presigned_url(
ClientMethod='get_object',
Params={
'Bucket': '...',
'Key': '{}.mp4'.format(video_id)
}
)
dictionary = {
'id': video_id,
'location': row[1],
'src': url
}
infos['videos'].append(dictionary)
return json.dumps(infos)
Thanks for any ideas.
Most of the time in your program is probably wasted by waiting for the network. Indeed you call s3.generate_presigned_url which will send a request to Amazon and then you have to wait until the server finally responds. In the meantime there is no much processing you can do.
So the most potential is to speed the process up by doing requests in parallel. So you send for instance 10 requests and then wait for the 10 responses. This article gives a brief introduction on this.
Based on your question, and the article, you can use something like the following to speed up the process:
from multiprocessing.pool import ThreadPool
# ...
def fetch_generate_presigned_url(video_id):
return s3.generate_presigned_url(
ClientMethod='get_object',
Params={
'Bucket': '...',
'Key': '{}.mp4'.format(video_id)
}
)
def convert_to_json(self, data):
pool = ThreadPool(processes=10)
urls = [row[0] for row in data]
video_ids = pool.map(fetch_generate_presigned_url,urls)
infos = {
'videos':[{'id': video_id,'location': row[1],'src': row[0]}
for vide_id,row in zip(video_ids,data)],
'total_count': len(data)
}
return json.dumps(infos)The number of processes, process=10 can be set higher to make the requests more parallel.