If I have a replica set with mongodb, than a primary server is receiving all the wirte/read operations and writing them to the server. The secondary server are reading the operations from the oplog and replicating them. Now I would like to analyze the data in mongodb replica set with spark-mongodb-connector. I can install a spark cluster on all three nodes and run analytics on it in memory. I understand that spark cluster has a master node where I have to submit the spark job for analytics, or spark streaming. Both are installed on an application server in tomcat.
now I need to choose a master node to submit the job from my tomcat app server to the spark cluster. Should the Primary Server be the Spark Master node? and than the driver of an application can connect to submit jobs on it?. What would be the Spark master in a sharded cluster?
It doesn't really matter which node is the Spark Master in your cluster. The Spark master will be responsible for assigning the tasks to the Spark executors, it will not receive all read/write requests. Each executor will then be responsible for fetching the data it needs to process.
Be careful about data partitioning in Spark, it might happen that mongoDB only provides a single partition to start with, so you might want to do a repartition first.