What I have.
A FastAPI app which consists of two sub apps and are mounted and specific paths. One of them is a websocket app. A single node Kafka cluster with two topics. One to which the websocket app writes and other from which the websocket app reads. A react application that uses socket.io client to make connection to the FastAPI app. Specifically the websocket sub app.
What I am trying to achieve
I am able to write to the Kafka topic from the websocket sub app as it is straightforward. Every user that connects to the app is given a socket it. I am mapping every socket id to the user identifier and storing it for later use. After the producer produces the message there is some other processing that is done on the data by another app and then it writes to a topic from which I have to read the messages. Basically a consumer. In the message I have a user identifier using which I can detect the socket id.
Now, I am unsure as to how to and when to start my consumer with respect to a FastAPI application. Does it need to be a standalone module? If yes, how do I send back the response to the original websocket client. [I have socket id stored.]
Firstly this is a common issue, because you've effectively got a consumer in a polling loop that is then publishing once it gets an event, can definitely impact the performance of the event-loop dependant on load. However, it's doable, assuming you use the rights bits and pieces.
In this example, you could re-tool the AIOKafka consumer to publish over the socket required. You'll have to maintain a mapping of socket ID to record ID (use redis).
BUT this is still adding a lot of things to a single fast API worker (assuming gunicon) and you'd need to watch consumer groups and partitions very carefully to make sure messages actually get to the right user.
Personally, I'd decouple this with a separate system to handle the realtime sending to the client, either with a hosted websocket pub-sub system, or using something like faust-streaming
Disclaimer: I wrote the linked blog, working at Ably.