How do I use persistent storage to save datasets and system state?

You can use the Lambda Cloud Storage feature to save:

• Large datasets that you don’t want to re-upload every time you start an instance
• The state of your system, including software packages and configurations

Preserving the state of your system

For saving the state of your system, including:

• Packages installed system-wide using apt-get
• Python packages installed using pip
• conda environments

We recommend creating containers using Docker or other software for creating containers.

You can also create a script that runs the commands needed to re-create your system state. For example:

sudo apt install PACKAGE_0 PACKAGE_1 PACKAGE_2 && \
pip install PACKAGE_3 PACKAGE_4 PACKAGE_5

Run the script each time you start an instance.

If you only need to preserve Python packages and not packages installed system-wide, you can create a Python virtual environment.

You can also create a conda environment.