I need to convert some of my Django views to work with async functions that query data sources. I'm experiencing big performance issues as those queries are executed one by one in series. However, the task is much harder than anticipated.
I've indicated below where the problems start. I'm experiencing other problems as well, however, this is by far the one that I don't have a clue on what to do. I get the following error where indicated in the code below:
django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async
model2 is a ForeignKey property pointing to another Model.
Wrapping model1.model2 inside sync_to_async() does not work.
Any idea how to make this work ?
async def queryFunctionAsync(param1, param2, loop):
model1 = await sync_to_async(Model1.objects.get)(pk=param1)
model2 = model1.model2 # This is where the error is generated
def exampleView(request):
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
data = async_to_sync(queryFunctionAsync)(param1, param2, loop)
loop.close()
Assuming you have some kind of Django view and when you make a request to the view you want a function to run in the background to improve the response time of the view. There isn't much point having async functions if you have to convert them to synchronous in order to make them compatible with your view or insert an await everywhere.
If it was me I would recommend using an event streaming framework like celery which will allow you to run tasks in the background and offload processing from your django views at scale. An example would be a a django view that allows the user to post a message to a forum, you might have some natural language processing that might occur after the message is sent. Instead of blocking your django view threads up you can offload the processing to a @shared_task in celery and this will improve the UX.
A worker in your celery pool will pick up the task from a queue and perform the processing in the background. After background processing has completed you can use django-channels websockets to alert the user of the completion of the task and any results that need presented to the client. Celery also allows you to chain multiple tasks together and wait for a result come back using a concept called a chord callback.
As your queries take longer and datasets increase in size you can also take advantage of flat file db services such as elasticsearch or mongodb to improve performance of more advanced nested querying at scale.