We are creating a simple Dockerfile, the last line of that file is
ENTRYPOINT ["sh", "-c", "spark-submit --master $SPARK_MASTER script.py"]
The script.py is a simple pyspark app (is not important for this discussion), this pyspark app receives some parameters that we are trying to pass using the docker command as follows
docker run --rm my_spark_app_image --param1 something --param2 something_else
But script.py is not getting any parameter, i.e. the container executed:
spark-submit --master $SPARK_MASTER script.py
The expected behaviour is that the container executes:
spark-submit --master $SPARK_MASTER script.py --param1 something --param2 something_else
What am I doing wrong?
The /bin/sh -c only takes one argument, the script to run. Everything after that argument is a a shell variable $0, $1, etc, that can be parsed by the script. While you could do this with the /bin/sh -c syntax, it's awkward and won't grow with you in the future.
Rather than trying to parse the variables there, I'd move this into an entrypoint.sh that you include in your image:
#!/bin/sh
exec spark-submit --master $SPARK_MASTER script.py "$@"
And then change the Dockerfile to define:
COPY entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
The exec syntax replaces the shell script in PID 1 with the spark-submit process, which allows signals to be passed through. The "$@" will pass through any arguments from docker run, with each arg quoted in case you have spaces in the parameters. And since it's run by a shell script, the $SPARK_MASTER will be expanded.