Java Spark bug with nested schema (gharchive 3.5GB)

[a10y]

Issue Description

aduffy@Andrews-MacBook-Pro ~/D/spark-3.5.7-bin-hadoop3 [SIGINT]> RUST_BACKTRACE=full ./bin/spark-sql --jars /Volumes/Code/vortex/java/vortex-spark/build/libs/vortex-spark-0.56.0-62-ga9680d2.dirty-all.jar
25/11/24 10:15:05 WARN Utils: Your hostname, Andrews-MacBook-Pro.local resolves to a loopback address: 127.0.0.1; using 192.168.50.180 instead (on interface en0)
25/11/24 10:15:05 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
25/11/24 10:15:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
25/11/24 10:15:06 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
25/11/24 10:15:06 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
25/11/24 10:15:06 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
25/11/24 10:15:06 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore aduffy@127.0.0.1
Spark Web UI available at http://192.168.50.180:4040
Spark master: local[*], Application Id: local-1763997305769
spark-sql (default)> CREATE OR REPLACE TEMPORARY VIEW events USING vortex OPTIONS (path '/Volumes/Code/vortex/bench-vortex/data/gharchive/vortex-file-compressed/events.vortex');
25/11/24 10:15:12 WARN SparkStringUtils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.
Time taken: 0.625 seconds
spark-sql (default)> select count(*) from events;
25/11/24 10:15:17 WARN GarbageCollectionMetrics: To enable non-built-in garbage collector(s) List(G1 Concurrent GC), users should configure it(them) to spark.eventLog.gcMetrics.youngGenerationGarbageCollectors or spark.eventLog.gcMetrics.oldGenerationGarbageCollectors
Exception in thread "vortex-prefetch-thread" java.lang.RuntimeException: Prefetching interrupted
        at dev.vortex.spark.read.PrefetchingIterator.prefetchLoop(PrefetchingIterator.java:50)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.InterruptedException
        at java.base/java.lang.Object.wait0(Native Method)
        at java.base/java.lang.Object.wait(Object.java:366)
        at java.base/java.lang.Object.wait(Object.java:339)
        at dev.vortex.spark.read.PrefetchingIterator.prefetchLoop(PrefetchingIterator.java:40)
        ... 1 more
25/11/24 10:15:26 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.IllegalArgumentException: should have as many children as in the schema: found 0 expected 6
        at dev.vortex.relocated.org.apache.arrow.util.Preconditions.checkArgument(Preconditions.java:255)
        at dev.vortex.relocated.org.apache.arrow.vector.VectorLoader.loadBuffers(VectorLoader.java:153)
        at dev.vortex.relocated.org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:88)
        at dev.vortex.relocated.org.apache.arrow.c.Data.importIntoVectorSchemaRoot(Data.java:348)
        at dev.vortex.jni.JNIArray.exportToArrow(JNIArray.java:51)
        at dev.vortex.spark.read.VortexColumnarBatchIterator.next(VortexColumnarBatchIterator.java:83)
        at dev.vortex.spark.read.VortexPartitionReader.get(VortexPartitionReader.java:43)
        at dev.vortex.spark.read.VortexPartitionReader.get(VortexPartitionReader.java:22)
        at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.next(DataSourceRDD.scala:161)
        at org.apache.spark.sql.execution.datasources.v2.MetricsBatchIterator.next(DataSourceRDD.scala:207)
        at org.apache.spark.sql.execution.datasources.v2.MetricsBatchIterator.next(DataSourceRDD.scala:204)
        at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.next(DataSourceRDD.scala:75)
        at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:40)
        at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.hashAgg_doAggregateWithoutKey_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
        at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
        at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
        at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
        at org.apache.spark.scheduler.Task.run(Task.scala:141)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)                                                                                                                                               at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
25/11/24 10:15:26 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0) (192.168.50.180 executor driver): java.lang.IllegalArgumentException: should have as many children as in the schema: found 0 expected 6
        at dev.vortex.relocated.org.apache.arrow.util.Preconditions.checkArgument(Preconditions.java:255)
        at dev.vortex.relocated.org.apache.arrow.vector.VectorLoader.loadBuffers(VectorLoader.java:153)
        at dev.vortex.relocated.org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:88)                                                                                                                                at dev.vortex.relocated.org.apache.arrow.c.Data.importIntoVectorSchemaRoot(Data.java:348)
        at dev.vortex.jni.JNIArray.exportToArrow(JNIArray.java:51)
        at dev.vortex.spark.read.VortexColumnarBatchIterator.next(VortexColumnarBatchIterator.java:83)
        at dev.vortex.spark.read.VortexPartitionReader.get(VortexPartitionReader.java:43)
        at dev.vortex.spark.read.VortexPartitionReader.get(VortexPartitionReader.java:22)
        at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.next(DataSourceRDD.scala:161)
        at org.apache.spark.sql.execution.datasources.v2.MetricsBatchIterator.next(DataSourceRDD.scala:207)
        at org.apache.spark.sql.execution.datasources.v2.MetricsBatchIterator.next(DataSourceRDD.scala:204)
        at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.next(DataSourceRDD.scala:75)
        at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:40)                                                                                                                                         at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.hashAgg_doAggregateWithoutKey_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)                                                    at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
        at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
        at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
        at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
        at org.apache.spark.scheduler.Task.run(Task.scala:141)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)                                                                                                                                 at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)                                                                                                                                               at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)                                                                                                                           at java.base/java.lang.Thread.run(Thread.java:1583)

25/11/24 10:15:26 ERROR TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting job
Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (192.168.50.180 executor driver): java.lang.IllegalArgumentException: should have as many children as in the schema: found 0 expected 6
        at dev.vortex.relocated.org.apache.arrow.util.Preconditions.checkArgument(Preconditions.java:255)
        at dev.vortex.relocated.org.apache.arrow.vector.VectorLoader.loadBuffers(VectorLoader.java:153)
        at dev.vortex.relocated.org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:88)
        at dev.vortex.relocated.org.apache.arrow.c.Data.importIntoVectorSchemaRoot(Data.java:348)
        at dev.vortex.jni.JNIArray.exportToArrow(JNIArray.java:51)
        at dev.vortex.spark.read.VortexColumnarBatchIterator.next(VortexColumnarBatchIterator.java:83)                                                                                                                         at dev.vortex.spark.read.VortexPartitionReader.get(VortexPartitionReader.java:43)
        at dev.vortex.spark.read.VortexPartitionReader.get(VortexPartitionReader.java:22)
        at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.next(DataSourceRDD.scala:161)                                                                                                                       at org.apache.spark.sql.execution.datasources.v2.MetricsBatchIterator.next(DataSourceRDD.scala:207)
        at org.apache.spark.sql.execution.datasources.v2.MetricsBatchIterator.next(DataSourceRDD.scala:204)                                                                                                                    at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.next(DataSourceRDD.scala:75)
        at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:40)                                                                                                                                         at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.hashAgg_doAggregateWithoutKey_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
        at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
        at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
        at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
        at org.apache.spark.scheduler.Task.run(Task.scala:141)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)                                                                                                                                    at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)                                                                                                                                               at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
                                                                                                                                                                                                                       Driver stacktrace:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (192.168.50.180 executor driver): java.lang.IllegalArgumentException: should have as many children as in the schema: found 0 expected 6
        at dev.vortex.relocated.org.apache.arrow.util.Preconditions.checkArgument(Preconditions.java:255)
        at dev.vortex.relocated.org.apache.arrow.vector.VectorLoader.loadBuffers(VectorLoader.java:153)
        at dev.vortex.relocated.org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:88)
        at dev.vortex.relocated.org.apache.arrow.c.Data.importIntoVectorSchemaRoot(Data.java:348)
        at dev.vortex.jni.JNIArray.exportToArrow(JNIArray.java:51)
        at dev.vortex.spark.read.VortexColumnarBatchIterator.next(VortexColumnarBatchIterator.java:83)
        at dev.vortex.spark.read.VortexPartitionReader.get(VortexPartitionReader.java:43)
        at dev.vortex.spark.read.VortexPartitionReader.get(VortexPartitionReader.java:22)                                                                                                                                      at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.next(DataSourceRDD.scala:161)
        at org.apache.spark.sql.execution.datasources.v2.MetricsBatchIterator.next(DataSourceRDD.scala:207)
        at org.apache.spark.sql.execution.datasources.v2.MetricsBatchIterator.next(DataSourceRDD.scala:204)                                                                                                                    at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.next(DataSourceRDD.scala:75)
        at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:40)
        at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.hashAgg_doAggregateWithoutKey_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
        at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
        at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)                                                                                                                                at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)                                                                                                                                          at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
        at org.apache.spark.scheduler.Task.run(Task.scala:141)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2898)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2834)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2833)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2833)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1253)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1253)
        at scala.Option.foreach(Option.scala:407)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1253)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3102)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3036)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3025)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
Caused by: java.lang.IllegalArgumentException: should have as many children as in the schema: found 0 expected 6
        at dev.vortex.relocated.org.apache.arrow.util.Preconditions.checkArgument(Preconditions.java:255)
        at dev.vortex.relocated.org.apache.arrow.vector.VectorLoader.loadBuffers(VectorLoader.java:153)
        at dev.vortex.relocated.org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:88)
        at dev.vortex.relocated.org.apache.arrow.c.Data.importIntoVectorSchemaRoot(Data.java:348)
        at dev.vortex.jni.JNIArray.exportToArrow(JNIArray.java:51)
        at dev.vortex.spark.read.VortexColumnarBatchIterator.next(VortexColumnarBatchIterator.java:83)
        at dev.vortex.spark.read.VortexPartitionReader.get(VortexPartitionReader.java:43)
        at dev.vortex.spark.read.VortexPartitionReader.get(VortexPartitionReader.java:22)
        at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.next(DataSourceRDD.scala:161)
        at org.apache.spark.sql.execution.datasources.v2.MetricsBatchIterator.next(DataSourceRDD.scala:207)
        at org.apache.spark.sql.execution.datasources.v2.MetricsBatchIterator.next(DataSourceRDD.scala:204)
        at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.next(DataSourceRDD.scala:75)
        at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:40)
        at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.hashAgg_doAggregateWithoutKey_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
        at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
        at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
        at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
        at org.apache.spark.scheduler.Task.run(Task.scala:141)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
spark-sql (default)>

Expected Behavior

Count should execute correctly

Actual Behavior

Throws an exception. When I set a debugger, it's failing inside of the actor column

Reproduction Steps

Attempt to query the gharchive dataset with Spark. I'm wondering if this is a bug somewhere deep inside of the C Data Interface code for arrow-rs.

OS Version Information

macOS 26

I acknowledge that:

• I have searched the Vortex repository (both open and closed Discussions and Issues) and confirm this is not a duplicate of an existing issue or discussion.
• I have checked the "Preview" tab on all text fields to ensure that everything looks right, and have wrapped all configuration and code in code blocks with a group of three backticks (```) on separate lines.

[robert3005]

We have been refactored a lot at this point