Skip to content

`JJAVA_STARTUP_SCRIPT can't start up spark #67

New issue
New issue
Closed
Bug
#102
Closed
`JJAVA_STARTUP_SCRIPT can't start up spark#67
Bug
#102
Milestone
1.0-a7

Description

@clhedrick
clhedrick
opened on Jun 12, 2025

With ijava, the following will set up spark:

"IJAVA_STARTUP_SCRIPT":"import org.apache.spark.; import org.apache.spark.sql.; import org.apache.spark.api.java.; import org.apache.spark.ml.linalg.; import org.apache.spark.sql.types.; import org.apache.spark.mllib.random.; SparkConf conf = null; JavaSparkContext jsc = null; SparkSession spark = null; SparkContext sc = null; class initspark{ public static void init() { String appName = "app"; String master = "local[4]"; conf = new SparkConf().setAppName(appName).setMaster(master); jsc = new JavaSparkContext(conf); spark=SparkSession.builder().sparkContext(jsc.sc()).master(master).getOrCreate(); sc=jsc.sc(); } }; "

It doesn't work with jjava. I'm not clear why not.

The closest I could come is the following

"IJAVA_STARTUP_SCRIPT":"import org.apache.spark.; import org.apache.spark.sql.; import org.apache.spark.api.java.; import org.apache.spark.ml.linalg.; import org.apache.spark.sql.types.; import org.apache.spark.mllib.random.; SparkConf conf = null; JavaSparkContext jsc = null; SparkSession spark = null; SparkContext sc = null; class initspark{ public static void init() { String appName = "app"; String master = "local[4]"; conf = new SparkConf().setAppName(appName).setMaster(master); jsc = new JavaSparkContext(conf); spark=SparkSession.builder().sparkContext(jsc.sc()).master(master).getOrCreate(); sc=jsc.sc(); } }; "

the `user than has to type "initspark.init()"

Activity

  1. andrus commented on Jun 12, 2025

    @andrus
    andrus
    on Jun 12, 2025
    Contributor

    Hi there. Without checking the actual script, one thing that I noticed is the var name. The name of the variable should be JJAVA_STARTUP_SCRIPT per https://dflib.org/jjava/docs/1.x/#environment-vars .

  2. clhedrick commented on Jun 13, 2025

    @clhedrick
    clhedrick
    on Jun 13, 2025 via email
    Author
  3. andrus
    added the
    Bug
    issue type on Jun 14, 2025

  4. andrus commented on Jun 14, 2025

    @andrus
    andrus
    on Jun 14, 2025
    Contributor

    Actually I am able to reproduce this myself:

    1. Download an unpack Spark spark-3.5.6-bin-hadoop3
    2. Set classpath:
    export JJAVA_CLASSPATH='/path/to/spark-3.5.6-bin-hadoop3/jars/*.jar'
    1. Set the startup script:
    export JJAVA_STARTUP_SCRIPT='
import org.apache.spark.api.java.*;
import org.apache.spark.sql.*;
import org.apache.spark.ml.linalg.*;
import org.apache.spark.sql.types.*;
import org.apache.spark.mllib.random.*;

SparkConf conf = null;
JavaSparkContext jsc = null;
SparkSession spark = null;
SparkContext sc = null;
class initspark{
    public static void init() {
        String appName = "app";
        String master = "local[4]";
        conf = new SparkConf().setAppName(appName).setMaster(master);
        jsc = new JavaSparkContext(conf);
        spark=SparkSession.builder().sparkContext(jsc.sc()).master(master).getOrCreate();
        sc=jsc.sc();
    }
}'

    Start jupyter and open any Java notebook. The following exception is printed in the logs (failing code details became visible after I implemented #68)

    Exception in thread "main" org.dflib.jjava.execution.CompilationException: [snippet: [SparkConf conf = null;], status: REJECTED]
	at org.dflib.jjava.execution.CodeEvaluator.evalSingle(CodeEvaluator.java:150)
	at org.dflib.jjava.execution.CodeEvaluator.eval(CodeEvaluator.java:170)
	at org.dflib.jjava.execution.CodeEvaluator.init(CodeEvaluator.java:84)
	at org.dflib.jjava.execution.CodeEvaluator.eval(CodeEvaluator.java:163)
	at org.dflib.jjava.JavaKernel.evalRaw(JavaKernel.java:337)
	at org.dflib.jjava.JavaKernel.eval(JavaKernel.java:342)
	at org.dflib.jjava.JJavaExtension.install(JJavaExtension.java:28)
	at org.dflib.jjava.JavaKernel.handleExtensionLoading(JavaKernel.java:165)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.dflib.jjava.magics.MavenResolver.initImplicitExtensions(MavenResolver.java:119)
	at org.dflib.jjava.JavaKernel.<init>(JavaKernel.java:157)
	at org.dflib.jjava.JJava.main(JJava.java:99)

    When I unset JJAVA_STARTUP_SCRIPT and copy the script into the notebook cell, everything works. We'll investigate.

  5. clhedrick commented on Jun 14, 2025

    @clhedrick
    clhedrick
    on Jun 14, 2025 via email
    Author

  6. clhedrick commented on Jun 14, 2025

    @clhedrick
    clhedrick
    on Jun 14, 2025
    Author

    The following works for me:

    {
    "argv": [
        "/usr/lib/jvm/java-21-openjdk-amd64/bin/java",
        "-jar",
        "/common/system/venv/python3.12-jupyter/share/jupyter/kernels/jjspark/jjava-launcher.jar",
        "/common/system/venv/python3.12-jupyter/share/jupyter/kernels/jjspark/jjava.jar",
        "{connection_file}"
    ],
    "display_name": "Java (jjava), initspark.init() to set up Spark",
    "language": "java",
    "interrupt_mode": "message",
    "env": {
      "JJAVA_JVM_OPTS": "-Dio.netty.tryReflectionSetAccessible=true --add-exports java.base/sun.nio.ch=ALL-UNNAMED",
      "JAVA_HONE": "/usr/lib/jvm/java-21-openjdk-amd64/",
      "JJAVA_CLASSPATH": ".:/common/system/spark-4.0.0-bin-hadoop3/jars/*",
       "SPARK_HOME": "/common/system/spark-4.0.0-bin-hadoop3",
      "JJAVA_STARTUP_SCRIPT":"import org.apache.spark.*; import org.apache.spark.sql.*; import org.apache.spark.api.java.*; import org.apache.spark.ml.linalg.*; import org.apache.spark.sql.types.*; import org.apache.spark.mllib.random.*; SparkConf conf = null; JavaSparkContext jsc = null; SparkSession spark = null; SparkContext sc = null;  class initspark{ public static void init() {  String appName = \"app\"; String master = \"local[4]\"; conf = new SparkConf().setAppName(appName).setMaster(master);  jsc = new JavaSparkContext(conf);  spark=SparkSession.builder().sparkContext(jsc.sc()).master(master).getOrCreate();  sc=jsc.sc();  }  }; "
    },
    "metadata": {
	"url": "https://github.com/dflib/jjava"
    }
}

    However I have to type initspark.init(). (It also works as IJAVA_STARTUP_SCRIPT, which I didn't expect.)

    Variations of it that try to run the code during startup fail. Indeed the following fails:

    "JJAVA_STARTUP_SCRIPT":"import org.apache.spark.*; import org.apache.spark.sql.*; import org.apache.spark.api.java.*; import org.apache.spark.ml.linalg.*; import org.apache.spark.sql.types.*; import org.apache.spark.mllib.random.*; SparkConf conf = null; JavaSparkContext jsc = null; SparkSession spark = null; SparkContext sc = null; String appName = \"app\"; "

    appName ends up null.

  7. clhedrick commented on Jun 14, 2025

    @clhedrick
    clhedrick
    on Jun 14, 2025
    Author

    I don't know whether the JVM opts are still needed. They were in a previous version of Java and Spark.

  8. andrus
    closed this as completedin #102on Nov 8, 2025
  9. andrus
    added a commit that references this issue on Nov 8, 2025

    JJAVA_STARTUP_SCRIPT can't start up spark #67

    a0417ca
  10. andrus
    added this to the 1.0-a7 milestone on Nov 8, 2025

  11. andrus commented on Nov 8, 2025

    @andrus
    andrus
    on Nov 8, 2025
    Contributor

    I think we finally got to the bottom of it. The problem wasn't Spark-specific, but rather related to premature loading of the script from JJAVA_STARTUP_SCRIPT during kernel boot sequence. This got addressed per #102 and will be a part of JJava 1.0-a7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Bug

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions