To make Python environment setup in cloud instance easier, I have added a shell script that installs required packages and a conda virtual environment that contains frequently used python libraries in this repository.
But I recommend that you go through all the steps at least once before executing aforementioned shell script, just so you have an idea of what we are doing.
Advantages of using cloud instance
By using jupyter notebook on cloud, anyone can take benefits of professional grade hardware from their personal computer system. Rather than setting up a workstation or server and keeping it up to date with latest hardware as time passes, you can just pay for a virtual machine that suits your needs and get your work done easily.
Other than hardware to test our models, we also get storage for scripts and all the files containing data.
Some options for cloud computing are : Google Cloud Platform, Digital Ocean, and Amazon Web Service(AWS).
You are charged on the basis of hours. See AWS pricing here.
Tutorial
The Virtual machine I am using here has following specifications but the choices are completely arbitrary :
Amazon Machine Image (AMI) : Ubuntu Server 18.04 LTS
Instance Type : t2.micro
Physical Memory : 1GB
Storage : 8 GB (Amazon EBS)
CPU : Intel Xeon family
Step 1 : Make an AWS account from here
Step 2 : Sign in
Step 3 : Go to EC2 and Launch Instance
Step 4 : Select Ubuntu
Step 5 : Select t2.micro as instance type
Step 6 : Go to “Configure Security Groups” tab
Give group name and group description that you like .
Step 7 : Add a rule and enter the following values
Type: Custom TCP
Protocol : TCP [Set by Default]
Port Range : 8888
Source : Anywhere
Step 8 : Review & Launch
Step 9 : AWS will prompt you to create a new key pair or select a new one
NOTE : If you select an existing key pair, you need to have that “.pem” file on your local machine.
For this tutorial we will make a new key pair.
Download the key pair you’ve just made.
Step 10 : Click on Launch Instances.
Scroll down and click on View Instances.
Your instance will have status : running.
Step 11 : Copy the downloaded key pair to any directory you want and change access permissions for it.
I am making a new directory “notebooks” and copying key pair into it. I will also change its permissions to 400(read permission to root user only).
Use these commands in your bash shell or you can do it using GUI.
# Making a new directory "notebooks" and switching to it
mkdir notebooks && cd notebooks # Copying key pair into "notebooks" directory
cp ~/Downloads/for-tutorial.pem ./# Changing permissions for key pair
chmod 400 for-tutorial.pem
Step 12 : Right click on your running instance and click on Connect.
The example will have a command like
ssh -i "for-tutorial.pem" ubuntu@ec2-XX-XXX-XX-XX.XX-XXXX-X.compute.amazonaws.comRun that command on shell in your local machine.
Answer yes to any prompt by shell and you’ll be connected to terminal of your cloud instance.
You could notice that your terminal now identifies machine as
@ubuntu@ip-xxx-xx-xx-xx:~$ Step 13 : Now you have to do following at cloud instance :
1. Install bzip2, gcc, git, htop, screen, vim, and wget.
2. Download and install anaconda (or miniconda).
3. Add conda to PATH variable.
4. Install necessary python libraries from conda (in this tutorial : jupyter notebook and ipython)
All of these four steps are covered in following command
or you can download the bash script that does it all from here.
#Step 1
apt-get update
apt-get upgrade -y
apt-get install -y bzip2 gcc git htop screen vim wget
apt-get upgrade -y bash
apt-get clean#Step 2
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O Miniconda.sh
bash Miniconda.sh -b
rm -rf Miniconda.sh#Step 3
echo "export PATH=~/miniconda3/bin:$PATH" >> ~/.bashrc
source ~/.bashrc#Step 4
conda install -c anaconda notebook --yes
conda install ipython --yes
Step 14 : Generate an OpenSSL certificate for Jupyter Notebook
Run this command on the cloud instance terminal.
openssl req -x509 -nodes -days 365 -newkey rsa:1024 -out mykey.pem -keyout mykey.pemThe name of file after -out and -keyout flag should be same.
Shell will prompt you to enter some details about yourself, you can fill them or just leave them blank using “ . ” as input.
A file named “mykey.pem” will be generated.
Step 15 : Generate a password hash for jupyter notebook
Run the following command in cloud instance terminal
ipythonThis will execute an interactive session for ipython in the terminal, now import passwd() method from notebook.auth and generate hash for any string, here I used “mypassword” but you can use something more secure.
In [1]: from notebook.auth import passwdIn [2]: passwd("mypassword")
Out[2]: 'sha1:553c099eea73:2e8fda20a9878dd5abc84965d4a81e09eb48995f'In [3]: exit
The string you generate hash for will be the password of jupyter notebook.
Do not clear the terminal window, you will need this hash in following steps.
Step 16 : Generate a Jupyter Notebook Configuration File (jupyter_notebook_config.py)
Run this in the terminal of cloud instance
jupyter notebook --generate-configStep 17 : Open config file (~/.jupyter/jupyter_notebook_config.py)
and using text editor (like vim or nano), add the following lines
c.get_config()
c.NotebookApp.certfile=u'home/ubuntu/mykey.pem'
c.NotebookApp.ip='0.0.0.0'
c.NotebookApp.port=8888
c.NotebookApp.password=u'sha1:553c099eea73:2e8fda20a9878dd5abc84965d4a81e09eb48995f'Here, c.NotebookApp.certfile takes the location of certificate generated by OpenSSL (mykey.pem)
and c.NotebookApp.password takes the hash generated from Step 15 as input.
Step 18 : Run jupyter notebook on cloud instance
jupyter notebook --no-browser --port=8888Step 19 : Open a new terminal tab or window on your local machine (keeping the existing jupyter notebook terminal open)
connect to cloud instance terminal but with a new flag -L
ssh -i "for-tutorial.pem" -L 8890:localhost:8888 ubuntu@ec2-XX-XXX-XX-XX.XX-XXXX-X.compute.amazonaws.comThis command will forward the output from 8888 port of cloud instance to port 8890 of your local machine.
Step 20 : Open localhost:8890 in your browser on you local machine and enter the password (string from step 15).
NOTE : If your browser warns you about an insecure connection, you can just choose to “continue anyway”.
If password is said to be incorrect, then copy the token from terminal that is running jupyter notebook and paste it in password field.
If something goes wrong, relaunch jupyter notebook on a different port and use a different port on your local machine as well.
