HPC Jupyter Kernels
To create a custom Jupyter kernel from jupyter.sdcc.bnl.gov for the HPC, You need terminal shell access. You can use ssh login to icsubmit01/02 via ssh gateway. Or just start a terminal from Jupyter session.
The following command will create a Jupyter kernel which will be based on any python environment your terminal session is in, whether it's some base python distribution environment or an activated python virtual env.
python -m ipykernel install --user --name="your_kernel_name" --display-name="display-name"
Then in a short while, your Jupyter session will be able to see the new kernel available.
On HPC cluster we suggest use python from Anaconda distribution. You can enable it from icsubmit01/02 or Jupyter terminal session by:
module load python
Or more specific version
module load python/3.8-anaconda-2020-11
This python distribution have many packages installed include some version of tensorfow-gpu and pytorch etc.
You can also use other distribution of python like Intel's distribution or you own installed version of python. To use Intel distributed python use:
module load intel
If you need a specific version of certain package which is not in the existing base env, or if you need different sets of packages/versions for different projects, then you need to create new virtual environments and install your need packages yourself.
Example of create python virtual env with python3.7 and some packages of some specific version:
module load python conda create -n my-env python=3.7 pip ipykernel package1 package2=some-version ... source activate my-env conda install additional_package1 additional_package2 pip install packages_not_available_in_conda
Now you can run your code or submit slurm job under this new virtualenv. If you want , you can create a Jupyter kernel of this env to be used from Jupyter.sdcc.bnl.gov
sbatch your_slurm_batch_script python -m ipykernel install --user --name "my-kernel-name" --display-name "display-name-on-jupyter"
After finish job , you might want deactivate the env so you may use base python environment or activate a different virtualenv for a different project which need different sets and versions of packages