Jupyter notebook

Anh-Thi Dinh
SkillsData ScienceDeep LearningMachine LearningPythonTools
☝
If you use VSCode, you should use its Jupyter Notebook extension, it's quick, clean and very easy to use.

Installation

Jupyter notebook

1# BY PIP
2pip install --upgrade pip
3pip install --upgrade ipython jupyter
1# BY CONDA
2conda install ipython jupyter
Or read more in Python Installation.
If you meet error OSError: [Errno 99] Cannot assign requested address, try
1jupyter notebook --ip=127.0.0.1 --port=8080
2# or
3jupyter notebook --ip=127.0.0.1 --port=8080 --allow-root

Setting up a password

1# create a juputer notebook config file
2# it can be used for other settings
3# <https://jupyter-notebook.readthedocs.io/en/stable/public_server.html#prerequisite-a-notebook-configuration-file>
4jupyter notebook --generate-config
5
6# create a new password
7# note: sha1 cannot be reverted!!
8jupyter notebook password
Inside notebook:
1from notebook.auth import passwd
2passwd()
With docker
1# create a sha1 password
2# download file create_sha1.py from <https://github.com/dinhanhthi/scripts>
3# run ./create_sha1.py
4
5# docker-compose.yml
6environment:
7  - PASSWD='sha1:d03968479249:319e92302e68d601392918f011d6c9334493023f'
8
9# Dockerfile
10CMD /bin/bash -c 'jupyter lab --no-browser --allow-root --ip=0.0.0.0 --NotebookApp.password="$PASSWD" "$@"'

R with jupyter notebook

Read more here.
1# install jupyter
2sudo apt-get install libzmq3-dev libcurl4-openssl-dev libssl-dev jupyter-core jupyter-client
3
4# install R on linux
5sudo apt install r-base
6
7# R kernel for Jupyter Notebook
8R # enter R environnement
9# install R kernel
10install.packages(c('repr', 'IRdisplay', 'IRkernel'), type = 'source')
11# or
12install.packages(c('repr', 'IRkernel'), type = 'source')
13# make jupyter see r kernel
14IRkernel::installspec() # current user
15IRkernel::installspec(user = FALSE) # global
1# embedded R
2# use by cell magic %%R
3pip install rpy2
4
5# in a notebook
6%load_ext rpy2.ipython
7
8# then use
9%%R
10# R's codes

Jupyter notebook in VSCode

👉 Note: VSCode

Use the running kernel in VSCode

Suppose that you are running a kernel at
1http://localhost:8888/lab?token=85e5ed15a9d9024b358abe1b42e66e563502b66cd69b5d59
In VSCode → Open a notebook → Click Select Kernel on the top right of the notebook → Existing jupyter server → … → paste the above url and hit Enter.

Other tips

  • Running 2 tasks in the same cell TAKE LONGER TIME than running each on different cells.
  • Download a folder in jupyter notebook:
    • Inside notebook, use:
        • 1%%bash
2tar -czf archive.tar.gz foldername
    • Or using nbzip (only working on current server).

Check the info

1# function's info
2?<func-name>
1# function's shortcode
2??<func-name>
1# get the list of current variables
2whos
Check where command executed from (in your $path)?
1!type python
1python is /Users/thi/anaconda/envs/python3.6/bin/python

Multiline commands

1# Using '\\'
2df.columns = df.columns.str.replace('.', ' ')\\
3                       .str.replace('\\s+', ' ')\\
4                       .str.strip().str.upper()
You CANNOT put # comments at the end of each line break!

Hotkeys / Shortcuts

There are 2 modes: command mode (pres ESC to activate) and edit mode (Enter to activate). Below are the most useful ones (for me).
You can edit / add more shortcuts in Help > Edit Keyboard Shortcuts.

Jupyter notebook on remote server

Open jupyter notebook in local browser but the backend-server is on remote.
  • If jupyter server is already running on remote at http://192.168.0.155:9889
      • 1ssh -N -L localhost:9888:192.168.0.155:9899 <username-remote>@<remote-host> -p <port>
2# if there is no port, remove `-p <port>`
      Open browser: http://localhost:9888 (type password if needed).
  • If jupyter server is not running on remote yet,
      • 1# connect to remote
2ssh <username-remote>@<remote-host> -p <port>
3# if there is no port, remove `-p <port>`
      On remote,
      1# run juputer with custom port
2jupyter notebook --no-browser --port=9899
3
4# if there is error `OSError: [Errno 99] Cannot assign requested address`
5jupyter notebook --ip=0.0.0.0 --no-browser --port=9899
6
7# if there is error `Running as root is not recommended`
8jupyter notebook --ip=0.0.0.0 --no-browser --port=9899 --alow-root
      It's running and there are somethings like that,
      1http://127.0.0.1:9889/?token=717d9d276f0537a9...831793df6319ad389accd
      Open another terminal window and type,
      1ssh -N -L localhost:9888:localhost:9889 <username-remote>@<remote-host> -p <port>
2# if there is no port, remove `-p <port>`
3# there is nothing but it's running
      Open browser:
      1<http://localhost:9888/?token=717d9d276f0537a9...831793df6319ad389accd>
You can choose any port number you wanna instead of 9888 and 9889 (they can be the same), note that, you need to use a port number GREATER THAN 8000!

Install new python package inside Jupyter Notebook

Using conda (ref)
1# Install a conda package in the current Jupyter kernel
2import sys
3!conda install --yes --prefix {sys.prefix} numpy
4
5# DON'T DO THIS
6!conda install --yes numpy
Using pip
1# Install a pip package in the current Jupyter kernel
2import sys
3!{sys.executable} -m pip install numpy
4
5# DON'T DO THIS
6!pip install numpy
Check version and update/upgrade,
1!pip show pandas

Display dataframes side-by-side

1from IPython.display import display_html
2def display_side_by_side(*args):
3    html_str=''
4    for df in args:
5        html_str+=df.to_html()
6    display_html(html_str.replace('table','table style="display:inline; margin-right: 5px;"'),raw=True)
1display_side_by_side(df1,df2,df1)

Get previous outputs

1_ # previous output
2__ # second-to-last output
3___ # third-to-last output

Display 2 figures side-by-side markdown cell

Put below codes in the markdown cell of Jupyter Notebook.
1<tr>
2  <td> <img src="Nordic_trails.jpg" alt="Drawing" style="width: 250px;"/> </td>
3  <td> <img src="Nordic_trails.jpg" alt="Drawing" style="width: 250px;"/> </td>
4</tr>

Magic Functions

  • Check the full list (in examples) here or their docs here.
  • You can define your custom magic functions here.
Auto update the new updated modules (put at the beginning of the notebook)
1%load_ext autoreload
2%autoreload 2 # Reload all modules every time before executing
3
4%autoreload 0 # disable autoreloader
Check more settings of %autoreload here.
Show the plots inside the notebook:
1%matplotlib inline
Get the commands from 1 to 4:
1%history -n 1-4 # get commands 1 to 4
With the bash command line + and using also curl
👉 Note: REST API with cURL.
1%%bash -s $APP_NAME
2
3APP_NAME=$1
4
5cat > ./predictor/instances.json <<END
6{
7   "instances": [
8     {
9       "data": {
10         "b64": "WW91IGFyZW4ndCBraW5kLCBpIGhhdGUgeW91Lg=="
11       }
12     }
13   ]
14}
15END
16
17curl -s -X POST \\
18  -H "Content-Type: application/json; charset=utf-8" \\
19  -d @./predictor/instances.json \\
20  <http://localhost:7080/predictions/$APP_NAME/>
☝
We can run bash script inside a cell with ! pip install numpy.

Extensions

Table of contents

  1. Install npm and nodejs.
  1. Install this extension.
  1. Enable in jupyter lab view.
  1. Refresh the page.
1# errors
2# UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 23: ordinal not in range(128)
3npm config set unicode false

Debugger

  1. Install xeus-python, jupyterlab
      2. 1pip install xeus-python
2pip install jupyterlab
  1. Install this extension.
  1. Refresh the page, you have to choose kernel xpython (instead of Python 3) to use the debugger.

Convert notebook to HTML

1pip install jupyterlab
2jupyter nbconvert --to html <notebook>