Trying to read in csv files (spark.read.csv("s3://...)) stored in S3 using EMR. There are approximately 100 files in the S3 bucket, and ~300 instances in the EMR cluster each with two executors (7 cores each). I can see in the spark UI that nearly all tasks are allocated to just 10 executors, each with 10 tasks, rather than each task being assigned to different executors to take advantage of processing each file in parallel.

Is there a knob to force the tasks to distribute across all executors?

spark.dynamicAllocation.enabled is set to false, spark.executor.instances is set to the number of executors in the cluster.