How to get Spark running locally with notebooks

The goal here is show you how to set up your machine to run Spark using notebooks.

My motivation to do that was the fact that, a time ago, I did not know a lot around that, and when I first started to learn Spark I found hard to set up for just run a simple “hello world” sample. I still don’t know that much, but now I feel more comfortable to share this post, and hopefully I can help someone to get started smoothly than me at the first days :)

The first step is to install the pip packages ‘pyspark’ and ‘jupyter’ on the machine.

Also, as last step, we need to configure two others PySpark environment variables, so we can change the default behavior of ‘pyspark’ command.

First I’ve tried the docker image with the most recent version of all components:

It didn’t worked. So I tried with another version and then, bang! It worked just fine.

Then, to use the image, you just need run the following:

Just to keep note:

At the time of writing, I’ve just reinstalled the system due to an upgrade I’ve done on my notebook, and so I needed to configure everything again. So I just took the advantage that the steps were pretty fresh on my mind 😅

I tried to keep it simple, so that’s it :D

Thanks for your time reading.

Related posts:

Working and traveling abroad is not only a great experience, but can also kick start an international career and open doors to new opportunities worldwide. Your language skills will improve…

The flaw in this logic seemed too unbelievably obvious to be pointed out. Yet, when I heard this from my housemate, I made the mistake of trying to. I just couldn’t help it. Needless to say, my…

Flow sensors and flow meters play a crucial role in various industries and applications, making them essential components in many systems. These devices provide accurate and reliable measurement of…