I brew installed apache-spark via the homebrew
brew install apache-spark
Then ran
Spark-shell
And returned the following warnings:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/Cellar/apache-spark/3.2.0/libexec/jars/spark-unsafe_2.12-3.2.0.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/01/10 18:21:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://ip-192-168-1-176.ec2.internal:4040
Spark context available as 'sc' (master = local[*], app id = local-1641860464071).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.2.0
/_/
Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 11.0.12)
Type in expressions to have them evaluated.
Type :help for more information.
Here is my java --version
openjdk 11.0.12 2021-07-20
OpenJDK Runtime Environment Homebrew (build 11.0.12+0)
OpenJDK 64-Bit Server VM Homebrew (build 11.0.12+0, mixed mode)
How do I resolve the warnings?
There are two issues that need to be addressed to get rid of these errors:
Java11 and Spark 3.2 when used together produce these errors. The first step is to switch to Java8 until this issue gets resolved.
I get the same errors using brew that you do, even if I am using Java8. Instead of using brew, go to the Apache Website at https://www.apache.org/dyn/closer.lua/spark/spark-3.2.0/spark-3.2.0-bin-hadoop3.2.tgz and unzip this file. To set your PATH variable so that you can type "spark-shell", you'll need to go into your .bash, .profile, or .zshrc (whichever you use) and add "export SPARK_HOME=path-to-spark-folder" and then add $SPARK_HOME to your PATH variable.