How to setup Jupyter Notebook to run Scala and Spark?

Spark, Scala, Jupyter

If you are working in the Data Science domain, then you are already familiar with Jupyter Notebook. It’s one of the most popular interactive tools to develop ML projects in Python. But you can also configure your Jupyter Notebook to run Scala and Spark with some easy steps.

Steps to set Jupyter Notebook to run Scala and Spark.

  1. Prerequisites:

    1. Make sure that JRE is available in your machine and it’s added to the PATH environment variable. In my system Java Home is: C:\Program Files\Java\jre1.8.0_281
    2. Anaconda is installed in your system.
    3. If you just want to run Scala codes, then above prerequisites are enough. If you want to use Spark as well, then make sure that Spark is installed in your system.
    I’ll create a different tutorial on how to install Spark on a Windows system and will link that post here.

  2. Install spylon-kernal package

    Open your Anaconda Powershell and run the command below.

    pip install spylon-kernal

    install spylon-kernal

  3. Install Jupyter kernal

    Use the command below to install Jupyter kernel. If you face any permission issue, then re-launch the Anaconda Powershell as Administrator.

    python -m spylon_kernel install 

    spylon kernal scala

  4. Launch Jupyter Notebook

    Launch Jupyter notebook, then click on New and select spylon-kernel.

    run jupyter scala

  5. Run basic Scala codes

    You can see some of the basic Scala codes, running on Jupyter.

    scala code running on jupyter

  6. Spark with Scala code:

    Now, using Spark with Scala on Jupyter:

    spark with scala on jupyter

  7. Check Spark Web UI

    It can be seen that Spark Web UI is available on port 4041. In Web UI go to Environment tab and you can see the below details.

    Spark Web UI Env

Anik is an IT professional and Data Science Enthusiast. He loves to spend a lot of time testing and reviewing the latest gadgets and software. He likes all things tech and his passion for smartphones is only matched by his passion for Sci-Fi TV Series.