HPC Jupyter Kernels

To create a custom Jupyter kernel from jupyter.sdcc.bnl.gov for the HPC, You need  terminal shell access. You can  use ssh login to ic2submit01 via ssh gateway. Or just start a terminal from Jupyter session. 

The following command will create a  Jupyter kernel which will be based on any python environment  your  terminal session is in, whether it's some base python distribution environment or an activated python virtual env.    

python -m ipykernel install --user --name="your_kernel_name" --display-name="display-name"

Then in a short while, your Jupyter session will be able to see the new kernel available.

On HPC cluster we suggest use python from Anaconda distribution. You can enable it from icsubmit01/02 or Jupyter terminal session by:

module load python

Or more specific version 

module load python/3.11-anaconda-2023.07

This python distribution have many packages installed include some version of tensorfow-gpu  and pytorch etc. 

You  can also use other distribution of python  like  system included python  or you own installed version of python. 

When use anaconda distribution of python , If you need a specific version of certain package which is not in the existing base env, or if you need different sets of packages/versions for different projects,  then  you need to create new conda virtual environments and install your need packages yourself.

 Example of  create python virtual env  with python3.7 and some packages of some specific version:

module load python
conda create -n my-env python=3.7 pip ipykernel package1 package2=some-version ...
source activate my-env 
conda install additional_package1 additional_package2
pip install packages_not_available_in_conda

Now you can run your code  or submit slurm job under this new virtualenv.  If you want , you can create a Jupyter kernel of this env to be used from  Jupyter.sdcc.bnl.gov  

sbatch your_slurm_batch_script

python -m ipykernel install --user --name "my-kernel-name" --display-name "display-name-on-jupyter"

After finish job , you might want deactivate the env so you may use base python environment or activate  a different virtualenv  for a different project which need different sets and versions of packages

conda deactivate

If you are not on our HPC system ( python module not available) or not using conda distribution of python.  It is suggested to also create python virtual environments for different projects unless you have only one project which you can use following to install additional package to your home directory area.

 pip install --user your_additional_packages

To create python virtual env, activate/deactivate it , and install package in virtual env please follow this link https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/

Again when you are in an activated  python virtual env , make your you have  ipykernel package installed and use this command to create Jupyter kernel

python -m ipykernel install --user --name="your_kernel_name" --display-name="display-name"