I have a container (a machine learning application) which is capable of loading pre-trained ml-models stored in a persistent volume. I can ask the application to load a particular model by giving its name via its REST API.
Now I want to scale up this application so that, I can load whatever the model, in any of the replicas (not in all of them) and should be able to parse data from that model.
I know this can be done by having multiple deployments and multiple services pointing to each of those deployments so that each instance will have a separate node port and I can access the REST API of each instance through these different node ports. By doing this, I can keep a record of which model is loaded on which instance in my own.
Is there any recommended way to accomplish this requirement without having multiple deployments but through replicas? (Like maintaining a single deployment file with replicas and a manual load balancer at the service level)
As you mentioned, the preferred way to achieve this is through multiple deployments and services.
Unless created manually, replicasets are managed by the deployment and you won't be able to have a single deployment with multiple replicasets running different versions.
It will definitely be easier for you to have one service + deployment per version and a single ingress in front to route the traffic based on some piece of information. It could be a header, sub-domain, path, etc.
To generate all your deployments and services, you could have a look at kustomize.