By William Streck… | Thu, 04/29/2021 - 15:30


HPC Jupyter Kernels

To create a custom Jupyter kernel from for the HPC, You need  terminal shell access. You can  use ssh login to icsubmit01/02 via ssh gateway. Or just start a terminal from Jupyter session. 

The following command will create a  Jupyter kernel which will be based on any python environment  your  terminal session is in, whether it's some base python distribution environment or an activated python virtual env.    

python -m ipykernel install --user --name="your_kernel_name" --display-name="display-name"

Then in a short while, your Jupyter session will be able to see the new kernel available.

On HPC cluster we suggest use python from Anaconda distribution. You can enable it from icsubmit01/02 or Jupyter terminal session by:

module load python

Or more specific version 

module load python/3.8-anaconda-2020-11

This python distribution have many packages installed include some version of tensorfow-gpu  and pytorch etc. 

You  can also use other distribution of python  like  Intel's  distribution or you own installed version of python. To use Intel distributed python use:

module load intel

If you need a specific version of certain package which is not in the existing base env, or if you need different sets of packages/versions for different projects,  then  you need to create new virtual environments and install your need packages yourself.

 Example of  create python virtual env  with python3.7 and some packages of some specific version:

module load python
conda create -n my-env python=3.7 pip ipykernel package1 package2=some-version ...
source activate my-env 
conda install additional_package1 additional_package2
pip install packages_not_available_in_conda

Now you can run your code  or submit slurm job under this new virtualenv.  If you want , you can create a Jupyter kernel of this env to be used from  

sbatch your_slurm_batch_script
python -m ipykernel install --user --name "my-kernel-name" --display-name "display-name-on-jupyter"

After finish job , you might want deactivate the env so you may use base python environment or activate  a different virtualenv  for a different project which need different sets and versions of packages

conda deactivate