UnsatisfiedLinkError while reading the dataframe

[baratamavinash225]

Hi Team,
I am getting the below error while reading the dataframe.
java.lang.UnsatisfiedLinkError: 'long org.apache.arrow.dataset.file.JniWrapper.createOrcFileFormat(java.lang.String[])'

Here is the code snippet that i am running
spark.catalog.createTable("web_site", "arrow", Map("path" -> path, "originalFormat" -> "orc"))

val path = "file:///Users/avinashbaratam/Downloads/userdata1.parquet" val df = spark.read.option(ArrowOptions.KEY_ORIGINAL_FORMAT, "parquet").option(ArrowOptions.KEY_FILESYSTEM, "filesystem").format("arrow").load(path)

I have tried both Parquet & ORC, also I have tried both HDFS and local filesystem.

My Spark Shell command looks like below -

**spark-shell --driver-java-options "--add-exports java.base/sun.nio.ch=ALL-UNNAMED" \

--master spark://INMLW76VMD6N:7077
--conf spark.driver.extraClassPath=/usr/local/Cellar/apache-spark/3.3.0/libexec/jars/spark-arrow-datasource-standard-1.4.0-jar-with-dependencies.jar
--conf spark.executor.extraClassPath=/usr/local/Cellar/apache-spark/3.3.0/libexec/jars/spark-arrow-datasource-standard-1.4.0-jar-with-dependencies.jar
--jars /usr/local/Cellar/apache-spark/3.3.0/libexec/jars/arrow-vector-7.0.0.jar, /usr/local/Cellar/apache-spark/3.3.0/libexec/jars/arrow-memory-netty-7.0.0.jar, /usr/local/Cellar/apache-spark/3.3.0/libexec/jars/arrow-memory-core-7.0.0.jar, /usr/local/Cellar/apache-spark/3.3.0/libexec/jars/arrow-format-7.0.0.jar, /usr/local/Cellar/apache-spark/3.3.0/libexec/jars/arrow-dataset-9.0.0.jar
--num-executors 1
--total-executor-cores 1
--conf spark.driver.cores=1
--conf spark.executor.cores=1
--conf spark.executor.instances=1`**

Here is my entire StackTrace -

java.lang.UnsatisfiedLinkError: 'long org.apache.arrow.dataset.file.JniWrapper.createOrcFileFormat(java.lang.String[])'
at org.apache.arrow.dataset.file.JniWrapper.createOrcFileFormat(Native Method)
at org.apache.arrow.dataset.file.format.OrcFileFormat.createOrcFileFormat(OrcFileFormat.java:44)
at org.apache.arrow.dataset.file.format.OrcFileFormat.(OrcFileFormat.java:29)
at org.apache.arrow.dataset.file.format.OrcFileFormat.create(OrcFileFormat.java:35)
at com.intel.oap.spark.sql.execution.datasources.v2.arrow.ArrowUtils$.getFormat(ArrowUtils.scala:120)
at com.intel.oap.spark.sql.execution.datasources.v2.arrow.ArrowUtils$.makeArrowDiscovery(ArrowUtils.scala:76)
at com.intel.oap.spark.sql.execution.datasources.v2.arrow.ArrowUtils$.readSchema(ArrowUtils.scala:47)
at com.intel.oap.spark.sql.execution.datasources.v2.arrow.ArrowUtils$.readSchema(ArrowUtils.scala:60)
at com.intel.oap.spark.sql.execution.datasources.arrow.ArrowFileFormat.convert(ArrowFileFormat.scala:59)
at com.intel.oap.spark.sql.execution.datasources.arrow.ArrowFileFormat.inferSchema(ArrowFileFormat.scala:65)
at org.apache.spark.sql.execution.datasources.DataSource.$anonfun$getOrInferFileFormatSchema$11(DataSource.scala:210)
at scala.Option.orElse(Option.scala:447)
at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:207)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:411)
at org.apache.spark.sql.execution.command.CreateDataSourceTableCommand.run(createDataSourceTables.scala:79)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecu