Setting-up Persistent Environment

Download and install a custom conda installation via Miniconda to Amazon EBS volume

In the terminal of an existing notebook instance, create a .sh file using your preferred editor. Example:

vim custom-script.sh

Copy the contents of the on-create script into the .sh file. This script creates a new Conda environment in a custom Conda installation. This script also installs NumPy and Boto3 in the new Conda environment. Note: The notebook instance must have internet connectivity to download the Miniconda installer and ipykernel.

#!/bin/bash

set -e

# OVERVIEW

# This script installs a custom, persistent installation of conda on the Notebook Instance's EBS volume, and ensures

# that these custom environments are available as kernels in Jupyter.

# The on-create script downloads and installs a custom conda installation to the EBS volume via Miniconda. Any relevant

# packages can be installed here.

# 1. ipykernel is installed to ensure that the custom environment can be used as a Jupyter kernel

# 2. Ensure the Notebook Instance has internet connectivity to download the Miniconda installer

sudo -u ec2-user -i <<'EOF'

unset SUDO_UID

# Install a separate conda installation via Miniconda

WORKING_DIR=/home/ec2-user/SageMaker/custom-miniconda

mkdir -p "$WORKING_DIR"

wget https://repo.anaconda.com/miniconda/Miniconda3-4.6.14-Linux-x86_64.sh -O "$WORKING_DIR/miniconda.sh"

bash "$WORKING_DIR/miniconda.sh" -b -u -p "$WORKING_DIR/miniconda"

rm -rf "$WORKING_DIR/miniconda.sh"

# Create a custom conda environment

source "$WORKING_DIR/miniconda/bin/activate"

KERNEL_NAME="custom_python"

PYTHON="3.6"

conda create --yes --name "$KERNEL_NAME" python="$PYTHON"

conda activate "$KERNEL_NAME"

pip install --quiet ipykernel

# Customize these lines as necessary to install the required packages

conda install --yes numpy

pip install --quiet boto3

EOF

Mark the script as executable, and then run it. Example:

chmod +x custom-script.sh

./custom-script.sh

When the installation is complete, stop the notebook instance.

Saving custom environments using lifecycle config

On the stopped notebook instance, add the on-start script as a lifecycle configuration. This script makes the custom environment available as a kernel in Jupyter every time that you start the notebook instance.

#!/bin/bash

set -e

# OVERVIEW

# This script installs a custom, persistent installation of conda on the Notebook Instance's EBS volume, and ensures

# that these custom environments are available as kernels in Jupyter.

# The on-start script uses the custom conda environment created in the on-create script and uses the ipykernel package

# to add that as a kernel in Jupyter.

# For another example, see:

# https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-add-external.html#nbi-isolated-environment

sudo -u ec2-user -i <<'EOF'

unset SUDO_UID

WORKING_DIR=/home/ec2-user/SageMaker/custom-miniconda/

source "$WORKING_DIR/miniconda/bin/activate"

for env in $WORKING_DIR/miniconda/envs/*; do

BASENAME=$(basename "$env")

source activate "$BASENAME"

ln -s $env ~/anaconda3/envs/"$BASENAME"

#python -m ipykernel install --user --name "$BASENAME" --display-name "Custom ($BASENAME)"

done

# Optionally, uncomment these lines to disable SageMaker-provided Conda functionality.

# echo "c.EnvironmentKernelSpecManager.use_conda_directly = False" >> /home/ec2-user/.jupyter/jupyter_notebook_config.py

# rm /home/ec2-user/.condarc

EOF

echo "Restarting the Jupyter server.."

restart jupyter-server

Start the notebook instance, and then install your custom libraries in the custom environment. For example, to install pyarrow:

import sys

!conda install --yes --prefix {sys.prefix} -c conda-forge pyarrow

If you get an error message that says that you need to update Conda, run the following commands. Then, try installing the custom libraries again.

!conda install -p "/home/ec2-user/anaconda3" "conda>=4.8" --yes

!conda install -p "/home/ec2-user/SageMaker/custom-miniconda/miniconda" "conda>=4.8" --yes

If you stop and then start your notebook instance, your custom Conda environment and libraries are still available. You don't have to install them again.

Create a custom conda environment

In the terminal of an existing notebook instance, create a .sh file using your preferred editor. Example:

vim custom-script.sh

Copy the contents of the script below into the .sh file. This script creates a new Conda environment in a custom Conda installation. This script also installs NumPy and Boto3 in the new Conda environment. Note: Make sure to modify: KERNEL_NAME="custom_python"PYTHON="3.6"

#!/bin/bash

set -e

# OVERVIEW

# This script installs a custom, persistent installation of conda on the Notebook Instance's EBS volume, and ensures

# that these custom environments are available as kernels in Jupyter.

# The on-create script downloads and installs a custom conda installation to the EBS volume via Miniconda. Any relevant

# packages can be installed here.

# 1. ipykernel is installed to ensure that the custom environment can be used as a Jupyter kernel

# 2. Ensure the Notebook Instance has internet connectivity to download the Miniconda installer

sudo -u ec2-user -i <<'EOF'

unset SUDO_UID

# Create a custom conda environment

WORKING_DIR=/home/ec2-user/SageMaker/custom-miniconda

source "$WORKING_DIR/miniconda/bin/activate"

KERNEL_NAME="custom_python_p36"

PYTHON="3.6"

conda create --yes --name "$KERNEL_NAME" python="$PYTHON"

conda activate "$KERNEL_NAME"

pip install --quiet ipykernel

# Customize these lines as necessary to install the required packages

conda install --yes numpy

pip install --quiet boto3

EOF

Mark the script as executable, and then run it. Example:

chmod +x custom-script.sh

2./custom-script.sh

When the installation is complete, stop the notebook instance.

Export/install your local env packages

You can export the list of all the packages installed using

pip freeze > requirements.txt

Upload the requirements.txt file to AWS SageMaker to deploy the code by

#!/bin/bash

set -e

# OVERVIEW

# This script installs a custom list of python packages in requirements.txt

sudo -u ec2-user -i <<'EOF'

unset SUDO_UID

# Create a custom conda environment

WORKING_DIR=/home/ec2-user/SageMaker/custom-miniconda

source "$WORKING_DIR/miniconda/bin/activate"

KERNEL_NAME="custom_python_x"

conda activate "$KERNEL_NAME"

REQ_DIR=/home/ec2-user/SageMaker/Config/install_packages

pip install -r "$REQ_DIR/requirements.txt"

EOF

setup persistent enviroment: https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-add-external.html
install persistent libraries: https://aws.amazon.com/premiumsupport/knowledge-center/sagemaker-lifecycle-script-timeout/
on-creat/on-start script for life configuration setup: amazon-sagemaker-notebook-instance-lifecycle-config-samples/scripts/persistent-conda-ebs at master · aws-samples/amazon-sagemaker-notebook-instance-lifecycle-config-samples
Install external libraries in notebook instances: https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-add-external.html
install package on conda SageMaker: https://aws.amazon.com/premiumsupport/knowledge-center/sagemaker-python-package-conda/
managening conda env: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#removing-an-environment
conda channels: https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/channels.html

Google Sites

Report abuse