Setting-up Persistent Environment

Download and install a custom conda installation via Miniconda to Amazon EBS volume

  • In the terminal of an existing notebook instance, create a .sh file using your preferred editor. Example:

vim custom-script.sh

  • Copy the contents of the on-create script into the .sh file. This script creates a new Conda environment in a custom Conda installation. This script also installs NumPy and Boto3 in the new Conda environment. Note: The notebook instance must have internet connectivity to download the Miniconda installer and ipykernel.

#!/bin/bash

set -e

# OVERVIEW

# This script installs a custom, persistent installation of conda on the Notebook Instance's EBS volume, and ensures

# that these custom environments are available as kernels in Jupyter.

# The on-create script downloads and installs a custom conda installation to the EBS volume via Miniconda. Any relevant

# packages can be installed here.

# 1. ipykernel is installed to ensure that the custom environment can be used as a Jupyter kernel

# 2. Ensure the Notebook Instance has internet connectivity to download the Miniconda installer

sudo -u ec2-user -i <<'EOF'

unset SUDO_UID

# Install a separate conda installation via Miniconda

WORKING_DIR=/home/ec2-user/SageMaker/custom-miniconda

mkdir -p "$WORKING_DIR"

wget https://repo.anaconda.com/miniconda/Miniconda3-4.6.14-Linux-x86_64.sh -O "$WORKING_DIR/miniconda.sh"

bash "$WORKING_DIR/miniconda.sh" -b -u -p "$WORKING_DIR/miniconda"

rm -rf "$WORKING_DIR/miniconda.sh"

# Create a custom conda environment

source "$WORKING_DIR/miniconda/bin/activate"

KERNEL_NAME="custom_python"

PYTHON="3.6"

conda create --yes --name "$KERNEL_NAME" python="$PYTHON"

conda activate "$KERNEL_NAME"

pip install --quiet ipykernel

# Customize these lines as necessary to install the required packages

conda install --yes numpy

pip install --quiet boto3

EOF

  • Mark the script as executable, and then run it. Example:

chmod +x custom-script.sh

./custom-script.sh

  • When the installation is complete, stop the notebook instance.

Saving custom environments using lifecycle config

  • On the stopped notebook instance, add the on-start script as a lifecycle configuration. This script makes the custom environment available as a kernel in Jupyter every time that you start the notebook instance.

#!/bin/bash


set -e


# OVERVIEW

# This script installs a custom, persistent installation of conda on the Notebook Instance's EBS volume, and ensures

# that these custom environments are available as kernels in Jupyter.

#

# The on-start script uses the custom conda environment created in the on-create script and uses the ipykernel package

# to add that as a kernel in Jupyter.

#

# For another example, see:

# https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-add-external.html#nbi-isolated-environment


sudo -u ec2-user -i <<'EOF'

unset SUDO_UID


WORKING_DIR=/home/ec2-user/SageMaker/custom-miniconda/

source "$WORKING_DIR/miniconda/bin/activate"


for env in $WORKING_DIR/miniconda/envs/*; do

BASENAME=$(basename "$env")

source activate "$BASENAME"

ln -s $env ~/anaconda3/envs/"$BASENAME"

#python -m ipykernel install --user --name "$BASENAME" --display-name "Custom ($BASENAME)"

done


# Optionally, uncomment these lines to disable SageMaker-provided Conda functionality.

# echo "c.EnvironmentKernelSpecManager.use_conda_directly = False" >> /home/ec2-user/.jupyter/jupyter_notebook_config.py

# rm /home/ec2-user/.condarc

EOF


echo "Restarting the Jupyter server.."

restart jupyter-server


  • Start the notebook instance, and then install your custom libraries in the custom environment. For example, to install pyarrow:

import sys

!conda install --yes --prefix {sys.prefix} -c conda-forge pyarrow

  • If you get an error message that says that you need to update Conda, run the following commands. Then, try installing the custom libraries again.

!conda install -p "/home/ec2-user/anaconda3" "conda>=4.8" --yes

!conda install -p "/home/ec2-user/SageMaker/custom-miniconda/miniconda" "conda>=4.8" --yes

  • If you stop and then start your notebook instance, your custom Conda environment and libraries are still available. You don't have to install them again.

Create a custom conda environment

  • In the terminal of an existing notebook instance, create a .sh file using your preferred editor. Example:

vim custom-script.sh

  • Copy the contents of the script below into the .sh file. This script creates a new Conda environment in a custom Conda installation. This script also installs NumPy and Boto3 in the new Conda environment. Note: Make sure to modify: KERNEL_NAME="custom_python"PYTHON="3.6"

#!/bin/bash


set -e


# OVERVIEW

# This script installs a custom, persistent installation of conda on the Notebook Instance's EBS volume, and ensures

# that these custom environments are available as kernels in Jupyter.

#

# The on-create script downloads and installs a custom conda installation to the EBS volume via Miniconda. Any relevant

# packages can be installed here.

# 1. ipykernel is installed to ensure that the custom environment can be used as a Jupyter kernel

# 2. Ensure the Notebook Instance has internet connectivity to download the Miniconda installer



sudo -u ec2-user -i <<'EOF'

unset SUDO_UID


# Create a custom conda environment

WORKING_DIR=/home/ec2-user/SageMaker/custom-miniconda

source "$WORKING_DIR/miniconda/bin/activate"

KERNEL_NAME="custom_python_p36"

PYTHON="3.6"


conda create --yes --name "$KERNEL_NAME" python="$PYTHON"

conda activate "$KERNEL_NAME"


pip install --quiet ipykernel


# Customize these lines as necessary to install the required packages

conda install --yes numpy

pip install --quiet boto3

EOF

  • Mark the script as executable, and then run it. Example:

chmod +x custom-script.sh

2./custom-script.sh

  • When the installation is complete, stop the notebook instance.

Export/install your local env packages

  • You can export the list of all the packages installed using

pip freeze > requirements.txt

  • Upload the requirements.txt file to AWS SageMaker to deploy the code by

#!/bin/bash

set -e

# OVERVIEW

# This script installs a custom list of python packages in requirements.txt

sudo -u ec2-user -i <<'EOF'

unset SUDO_UID

# Create a custom conda environment

WORKING_DIR=/home/ec2-user/SageMaker/custom-miniconda

source "$WORKING_DIR/miniconda/bin/activate"

KERNEL_NAME="custom_python_x"

conda activate "$KERNEL_NAME"

REQ_DIR=/home/ec2-user/SageMaker/Config/install_packages

pip install -r "$REQ_DIR/requirements.txt"

EOF