Question

0

112

Views

Adding jars to a Spark Job - spark-submit

Following is my script to kick off my spark job

#!/bin/bash
APP_DIR="/home/arvind/myApp"
JARS=$(echo $APP_DIR/lib/* | tr ' ' ',')
/home/arvind/spark3/bin/spark-submit \
  --master spark://server4:7078 \
  --verbose \
  --jars $JARS \
  --driver-class-path $APP_DIR/conf \
  --class com.test.spark.MySparkApplication \
  --conf spark.driver.extraJavaOptions="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -Dlog4j.configuration=file:$APP_DIR/conf/log4j.properties" \
  --conf spark.driver.memory="32g" \
  --conf spark.cores.max="20" \
 $APP_DIR/myApp.jar > $APP_DIR/logs/output.err 2> $APP_DIR/logs/output.txt &

The lib folder contains all the jar files, packaged with the application, including postgres dependencies

[arvind@server4 myApp]$ ls lib/postgresql-42.1.4.jar
lib/postgresql-42.1.4.jar

When I run the application on a spark3 cluster, using the above script, I get the following exception

265 Caused by: java.lang.ClassNotFoundException: org.postgresql.ds.PGSimpleDataSource
266         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
267         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
268         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
269         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
270         at com.zaxxer.hikari.util.UtilityElf.createInstance(UtilityElf.java:77)
271         ... 25 more

But if I change the above script to submit it to a spark2.4.0 cluster, it works fine.

Could you please help me with this ?

over 4 years ago · Santiago Trujillo

1 answers

Answer question

0

Answer question

Find remote jobs

Accepted Answer · 2022-05-11T21:09:39.887Z

If you add the jar to the --driver-class-path like so it should work:


#!/bin/bash
APP_DIR="/home/arvind/myApp"
JARS=$(echo $APP_DIR/lib/* | tr ' ' ',')
COLON_SEP_JARS=$(echo $APP_DIR/lib/* | tr ' ' ':')
/home/arvind/spark3/bin/spark-submit \
  --master spark://server4:7078 \
  --verbose \
  --jars $JARS \
  --driver-class-path $APP_DIR/conf:$COLON_SEP_JARS \
  --class com.test.spark.MySparkApplication \
  --conf spark.driver.extraJavaOptions="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -Dlog4j.configuration=file:$APP_DIR/conf/log4j.properties" \
  --conf spark.driver.memory="32g" \
  --conf spark.cores.max="20" \
 $APP_DIR/myApp.jar > $APP_DIR/logs/output.err 2> $APP_DIR/logs/output.txt &

0

112

Adding jars to a Spark Job - spark-submit

1 answers

0

Find remote jobs

Andres GPT