Skip to content

[CORE] Remove arrow relocation - #13096

Draft
marin-ma wants to merge 1 commit into
apache:mainfrom
marin-ma:remove-arrow-relocation
Draft

marin-ma wants to merge 1 commit into
apache:mainfrom
marin-ma:remove-arrow-relocation

Conversation

@marin-ma

@marin-ma marin-ma commented Sep 22, 2026

Copy link
Copy Markdown
Contributor
/opt/ibm/spark/python/lib/pyspark.zip/pyspark/sql/pandas/conversion.py:351: UserWarning: createDataFrame attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true; however, failed by the reason below:
  An error occurred while calling z:org.apache.spark.sql.api.python.PythonSQLUtils.readArrowStreamFromFile.
: java.lang.NoSuchMethodError: org/apache/arrow/vector/ipc/message/MessageMetadataResult.getMessage()Lorg/apache/arrow/flatbuf/Message; (loaded from file:/opt/gluten/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.8.0-2026-09-15-WXD242-RC1.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@805f93a6) called from class org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$2 (loaded from file:/opt/ibm/spark/jars/spark-sql_2.12-3.5.4.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@805f93a6).
	at org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$2.readNextBatch(ArrowConverters.scala:467)
	at org.apache.spark.sql.execution.arrow.ArrowConverters$$anon$2.<init>(ArrowConverters.scala:443)
	at org.apache.spark.sql.execution.arrow.ArrowConverters$.getBatchesFromStream(ArrowConverters.scala:442)
	at org.apache.spark.sql.execution.arrow.ArrowConverters$.$anonfun$readArrowStreamFromFile$2(ArrowConverters.scala:432)
	at org.apache.spark.util.SparkErrorUtils.tryWithResource(SparkErrorUtils.scala:48)
	at org.apache.spark.util.SparkErrorUtils.tryWithResource$(SparkErrorUtils.scala:46)
	at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:94)
	at org.apache.spark.sql.execution.arrow.ArrowConverters$.readArrowStreamFromFile(ArrowConverters.scala:430)
	at org.apache.spark.sql.api.python.PythonSQLUtils$.readArrowStreamFromFile(PythonSQLUtils.scala:96)
	at org.apache.spark.sql.api.python.PythonSQLUtils.readArrowStreamFromFile(PythonSQLUtils.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:575)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.base/java.lang.Thread.run(Thread.java:854)

@github-actions github-actions Bot added the CORE works for Gluten Core label Sep 22, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant