By William Streck… |

Managing Jupyter for Experiments

There are methods to customize both jupyter kernels listed in a running session (based on the $JUPYTER_PATH environment variable) and the actual jupyterlab sessions themselves both based on the user's group. Launching in a custom jupyterlab environment allows the different experiments that SDCC supports to maintain their own jupyterlab extensions and plugins. Launching a custom kernel allows the running of experiment-provided setup scripts to prepare the environment in the notebook with your software.

Custom Kernels

Currently SDCC provides a wide variety of kernels, many of which are quite out-dated (as of Fall 2021). We will try to provide a few working generic environments but will migrate the experiment-specific ones to the new area.

Paths

We will provide folders under /u0b/software/jupyter/$name which will be added to the $JUPYTER_PATH and you will be able to put custom kernels there.

Kernel Spec

In a folder called "kernels" under the experiment path above, put a folder of your choice there and in it you can put a file "kernel.json" there (and image files called logo-32x32.png or logo-64x64.png to have a custom logo). In kernel.json you put the argv, display-name, and language of launching that kernel.

Example

For example your experiment may have a setup script in /cvmfs/foo/bin/setup.sh that you source. You can create a file that sources that script and then calls python with the rest of your arguments, for example in /cvmfs/foo/bin/kernel_launch.sh like:

#!/bin/bash
source /cvmfs/foo/bin/setup.sh
python "$@"

And then in the kernel.json put:

{
"argv": [
 "/cvmfs/foo/bin/kernel_launch.sh",
 "-m",
 "ipykernel_launcher",
 "-f",
 "{connection_file}"
],
"display_name": "Python (foo experiment)",
"language": "python"
}

More documentation available here

Custom Jupyterlab

If you desire your own jupyterlab environment be used (allowing custom plugins and more) you can have such an option displayed to users in your experiment.

On the menu screen of the jupyter launcher you will see a place to select an environment:

select jupyterlab radio button

These buttons are presented based on group-ids the user is in. Each one reflects a path to a different python virtual environment with jupyterlab and notebook software installed. Please reach out to SDCC about creating a new area for your experiment and we can help you install software there. CVMFS should be used for HTC nodes whereas NFS/GPFS can be used for the HPC environment