Setting-up Persistent Environment
Download and install a custom conda installation via Miniconda to Amazon EBS volume
In the terminal of an existing notebook instance, create a .sh file using your preferred editor. Example:
vim custom-script.sh
Copy the contents of the on-create script into the .sh file. This script creates a new Conda environment in a custom Conda installation. This script also installs NumPy and Boto3 in the new Conda environment. Note: The notebook instance must have internet connectivity to download the Miniconda installer and ipykernel.
#!/bin/bash
set -e
# OVERVIEW
# This script installs a custom, persistent installation of conda on the Notebook Instance's EBS volume, and ensures
# that these custom environments are available as kernels in Jupyter.
# The on-create script downloads and installs a custom conda installation to the EBS volume via Miniconda. Any relevant
# packages can be installed here.
# 1. ipykernel is installed to ensure that the custom environment can be used as a Jupyter kernel
# 2. Ensure the Notebook Instance has internet connectivity to download the Miniconda installer
sudo -u ec2-user -i <<'EOF'
unset SUDO_UID
# Install a separate conda installation via Miniconda
WORKING_DIR=/home/ec2-user/SageMaker/custom-miniconda
mkdir -p "$WORKING_DIR"
wget https://repo.anaconda.com/miniconda/Miniconda3-4.6.14-Linux-x86_64.sh -O "$WORKING_DIR/miniconda.sh"
bash "$WORKING_DIR/miniconda.sh" -b -u -p "$WORKING_DIR/miniconda"
rm -rf "$WORKING_DIR/miniconda.sh"
# Create a custom conda environment
source "$WORKING_DIR/miniconda/bin/activate"
KERNEL_NAME="custom_python"
PYTHON="3.6"
conda create --yes --name "$KERNEL_NAME" python="$PYTHON"
conda activate "$KERNEL_NAME"
pip install --quiet ipykernel
# Customize these lines as necessary to install the required packages
conda install --yes numpy
pip install --quiet boto3
EOF
Mark the script as executable, and then run it. Example:
chmod +x custom-script.sh
./custom-script.sh
When the installation is complete, stop the notebook instance.
Saving custom environments using lifecycle config
On the stopped notebook instance, add the on-start script as a lifecycle configuration. This script makes the custom environment available as a kernel in Jupyter every time that you start the notebook instance.
#!/bin/bash
set -e
# OVERVIEW
# This script installs a custom, persistent installation of conda on the Notebook Instance's EBS volume, and ensures
# that these custom environments are available as kernels in Jupyter.
#
# The on-start script uses the custom conda environment created in the on-create script and uses the ipykernel package
# to add that as a kernel in Jupyter.
#
# For another example, see:
# https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-add-external.html#nbi-isolated-environment
sudo -u ec2-user -i <<'EOF'
unset SUDO_UID
WORKING_DIR=/home/ec2-user/SageMaker/custom-miniconda/
source "$WORKING_DIR/miniconda/bin/activate"
for env in $WORKING_DIR/miniconda/envs/*; do
BASENAME=$(basename "$env")
source activate "$BASENAME"
ln -s $env ~/anaconda3/envs/"$BASENAME"
#python -m ipykernel install --user --name "$BASENAME" --display-name "Custom ($BASENAME)"
done
# Optionally, uncomment these lines to disable SageMaker-provided Conda functionality.
# echo "c.EnvironmentKernelSpecManager.use_conda_directly = False" >> /home/ec2-user/.jupyter/jupyter_notebook_config.py
# rm /home/ec2-user/.condarc
EOF
echo "Restarting the Jupyter server.."
restart jupyter-server
Start the notebook instance, and then install your custom libraries in the custom environment. For example, to install pyarrow:
import sys
!conda install --yes --prefix {sys.prefix} -c conda-forge pyarrow
If you get an error message that says that you need to update Conda, run the following commands. Then, try installing the custom libraries again.
!conda install -p "/home/ec2-user/anaconda3" "conda>=4.8" --yes
!conda install -p "/home/ec2-user/SageMaker/custom-miniconda/miniconda" "conda>=4.8" --yes
If you stop and then start your notebook instance, your custom Conda environment and libraries are still available. You don't have to install them again.
Create a custom conda environment
In the terminal of an existing notebook instance, create a .sh file using your preferred editor. Example:
vim custom-script.sh
Copy the contents of the script below into the .sh file. This script creates a new Conda environment in a custom Conda installation. This script also installs NumPy and Boto3 in the new Conda environment. Note: Make sure to modify: KERNEL_NAME="custom_python"PYTHON="3.6"
#!/bin/bash
set -e
# OVERVIEW
# This script installs a custom, persistent installation of conda on the Notebook Instance's EBS volume, and ensures
# that these custom environments are available as kernels in Jupyter.
#
# The on-create script downloads and installs a custom conda installation to the EBS volume via Miniconda. Any relevant
# packages can be installed here.
# 1. ipykernel is installed to ensure that the custom environment can be used as a Jupyter kernel
# 2. Ensure the Notebook Instance has internet connectivity to download the Miniconda installer
sudo -u ec2-user -i <<'EOF'
unset SUDO_UID
# Create a custom conda environment
WORKING_DIR=/home/ec2-user/SageMaker/custom-miniconda
source "$WORKING_DIR/miniconda/bin/activate"
KERNEL_NAME="custom_python_p36"
PYTHON="3.6"
conda create --yes --name "$KERNEL_NAME" python="$PYTHON"
conda activate "$KERNEL_NAME"
pip install --quiet ipykernel
# Customize these lines as necessary to install the required packages
conda install --yes numpy
pip install --quiet boto3
EOF
Mark the script as executable, and then run it. Example:
chmod +x custom-script.sh
2./custom-script.sh
When the installation is complete, stop the notebook instance.
Export/install your local env packages
You can export the list of all the packages installed using
pip freeze > requirements.txt
Upload the requirements.txt file to AWS SageMaker to deploy the code by
#!/bin/bash
set -e
# OVERVIEW
# This script installs a custom list of python packages in requirements.txt
sudo -u ec2-user -i <<'EOF'
unset SUDO_UID
# Create a custom conda environment
WORKING_DIR=/home/ec2-user/SageMaker/custom-miniconda
source "$WORKING_DIR/miniconda/bin/activate"
KERNEL_NAME="custom_python_x"
conda activate "$KERNEL_NAME"
REQ_DIR=/home/ec2-user/SageMaker/Config/install_packages
pip install -r "$REQ_DIR/requirements.txt"
EOF
<Reference>
setup persistent enviroment: https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-add-external.html
install persistent libraries: https://aws.amazon.com/premiumsupport/knowledge-center/sagemaker-lifecycle-script-timeout/
on-creat/on-start script for life configuration setup: amazon-sagemaker-notebook-instance-lifecycle-config-samples/scripts/persistent-conda-ebs at master · aws-samples/amazon-sagemaker-notebook-instance-lifecycle-config-samples
Install external libraries in notebook instances: https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-add-external.html
install package on conda SageMaker: https://aws.amazon.com/premiumsupport/knowledge-center/sagemaker-python-package-conda/
managening conda env: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#removing-an-environment
conda channels: https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/channels.html