I have a fastapi application that is very heavy on multi-threaded computation. That is, some requests, when processed, creating a lot of new threads for the process to complete. These requests return an "ok" fast, but then continue to process in the background.

In addition, I get lots of requests for status checking (e.g. is the previous process has finished, and "get me the results of the previous process"). some of the "get results" requests create new connections to an external database.

This seem to create an overload of requests, and many of them get timed-out.

I want to be able somehow to "split" my thread pool, so that there always be a"bandwidth" for status check and get results (with its db calling), while there be enough "threads" for the processing to continue in the background without anything getting stuck.

I looked at request rate limiting, but it doesn't seem to address all the issues (it just refuses some requests instead of timing them out). I also looked at --limit-concurrency option of uvicorn, but am not sure what would be its effect on the number of workers/threads that my application uses behind the scene.

Any insight/reference would be welcome