I've created a python main application main.py, which I invoke with uvicorn main.main --reload. Which of course runs the following code...
if __name__ == '__main__':
main()
That part of the application runs constantly, reads data an processes it until the application is aborted manually. I use asyncio to run coroutines.
Task
I would like to build a small html dashboard on it, which can display the data that is constantly computed.
Question
How can I run these background calculations of main.py and still implement a dashboard/website with fastapi and jinja2?
What I have achieved so far
I can run the main application without any fastapi code. And I can run the dashboard without the background tasks. Both work fine independently. But fastapi does not run, when I add its code to the main application with the background computation. (How could it?!? I can only invoke either the main application or the fastapi app.)
Any architectural concepts are appreciated. Thank you.
Fastapi doesn't run because it cant be reached by python interpreter untill it complete your computations. You should start your web app independently of the main process, I strongly recommend you to use docker-compose.
As fastapi recommends you, you should use Dramatiq or Celery for huge background tasks, or you can just run separate service in compose services, for example:
# background.py
if __name__ == '__main__':
main()
# main.py
app = FastAPI()
docker-compose.yml:
services:
web-app-interface:
command: uvicorn main.main ...
my-daemon:
command: python background.py
You can make them communicate with a message broker, such as RabbitMQ etc. And never use multiprocessing with uvicorn, it can cause process leak, bcz uvicorn rules it's own workers.
A good approch is to use the on_event decorator with startup. The only thing to remain is to use asyncio.create_task to invoke the background task. As long as you don't await it, it will not block and thus fastapi/uvicorn can continue to serve any http request.
my_service = MyService()
@app.on_event('startup')
async def service_tasks_startup():
"""Start all the non-blocking service tasks, which run in the background."""
asyncio.create_task(my_service.start_processing_data())
Also, with this said, any request can consume the data of this background service.
@app.get("/")
def root():
return my_service.value
Think of MyService as any class of your liking. Kafka consumption, computations, etc. Of course, the value is just an example attribute of the Class MyService.