I am trying to use spark on Kubernetes cluster (existing setup: emr + yarn). Our use case is to handle too many jobs including short lived ones (few seconds to 15 minutes). Also, we have peak hours when many workers needs to run to handle 100s of jobs running concurrently.
So what I want to achieve, running master and fixed few workers (say 5) all time and increase workers to 40-50 at peak time. Also, I will prefer to use dynamic allocation.
I am setting it as below
Master image (spark-master:X)
FROM <BASE spark 3.1 Image build using dev/make-distribution.sh -Pkubernetes in spark>
ENTRYPOINT ["/opt/spark/sbin/start-master.sh", "-p", "8081", "<A long running server command that can accept get traffic on 8080 to submit jobs>"]
Worker worker image (spark-worker:X)
FROM <BASE spark 3.1 Image build using dev/make-distribution.sh -Pkubernetes in spark>
ENTRYPOINT ["/opt/spark/sbin/start-worker.sh", "spark//spark-master:8081" ,"-p", "8081", "<A long running server command to keep up the worker>"]
Deplyments
apiVersion: apps/v1
kind: Deployment
metadata:
name: spark-master-server
spec:
replicas: 1
selector:
matchLabels:
component: spark-master-server
template:
metadata:
labels:
component: spark-master-server
spec:
containers:
- name: spark-master-server
image: spark-master:X
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8081
---
apiVersion: v1
kind: Service
metadata:
name: spark-master
spec:
type: ClusterIP
ports:
- port: 8081
targetPort: 8081
selector:
component: spark-master-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: spark-worker-instance
spec:
replicas: 3
selector:
matchLabels:
component: spark-worker-instance
template:
metadata:
labels:
component: spark-worker-instance
spec:
containers:
- name: spark-worker-server
image: spark-worker:X
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8081
Questions
Reason we are trying not create master and driver dynamically per job (as given in example - http://spark.apache.org/docs/latest/running-on-kubernetes.html) is that it may be an overhead for large no. of small jobs.
Is this setup recommended ?
Don't think so.
Dynamic Resource Allocation is a property of a single Spark application "to dynamically adjust the resources your application occupies based on the workload."
Dynamic Resource Allocation spans its resource requirements regardless of available nodes in a cluster. As long as there are resources available and a cluster manager could assign them to a Spark application these resources are free to go.
What you seem to be trying to set up is how to scale the cluster itself up and down. In your case it's Spark Standalone and although technically it's possible with ReplicaSets (just a guess) I've never heard any earlier attempts at it. You're on your own as Spark Standalone does not support it out of the box.
That I think is an overkill since you're building a multi-layer cluster environment: using a cluster manager (Kubernetes) to host another cluster manager (Spark Standalone) for Spark applications. Given Spark on Kubernetes supports Dynamic Allocation out of the box the only worry of yours should simply be how to "throw in" more CPU and memory on demand while resizing Kubernetes cluster. You should rely on the capabilities of Kubernetes to resize itself up and down rather than Spark Standalone on Kubernetes.
The spark on k8s operator may provide at least one mechanism to dynamically provision the resources you need to do safe scaling of resources based on demand.
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
My thinking is that instead of performing a direct spark submit to a master in a static spark cluster, you could make a call to the k8s API to provision the required instance; or otherwise define these as a cron schedule.
In the dynamic provisioning scenario one thing to consider is how your workloads get distributed across the cluster; we can’t use simple HPA rules for this as it may not be safe to tear down a worker on CPU/Mem levels; spawning a separate cluster for each on demand workload bypasses this but may not be optimal. I would be interested to hear how you get on.