I found that AWS Glue set up executor's instance with memory limit to 5 Gb --conf spark.executor.memory=5g and some times, on a big datasets it fails with java.lang.OutOfMemoryError. The same is for driver instance --spark.driver.memory=5g.
Is there any option to increase this value?
despite aws documentation stating that the --conf parameter should not be passed, our AWS support team told us to pass --conf spark.driver.memory=10g which corrected the issue we were having
You can override the parameters by editing the job and adding job parameters. The key and value I used are here:
Key: --conf
Value: spark.yarn.executor.memoryOverhead=7g
This seemed counterintuitive since the setting key is actually in the value, but it was recognized. So if you're attempting to set spark.yarn.executor.memory the following parameter would be appropriate:
Key: --conf
Value: spark.yarn.executor.memory=7g