Environment and dependency management in Python

Last reviewed

February 14, 2025

Last modified

March 25, 2025

When working with Python, managing dependencies and environments is important to ensure your project can be reproduced and shared.

Definitions:

A dependency is any external library your project needs, and a virtual environment is an isolated workspace where dependencies are installed.

There are several ways to manage dependencies and environments:

Conda Environments

Conda is a package and environment manager popular in the research and data science community. It allows you to manage both Python and non-Python dependencies.

Basic commands

# Create a new environment, e.g. with python 3.12
conda create -n your_env_name python=3.12

# List all environments
conda env list

# Activate an environment
conda activate your_env_name

# Install packages in an environment
conda install package_name

# Remove a package
conda remove package_name

# Export an environment to a file
conda env export > environment.yml

# Deactivate an environment
conda deactivate

# Remove an environment
conda env remove -n your_env_name

Conda environment files

Conda environment files (environment.yml) are used to specify the dependencies of a project. They can be used to create an environment from scratch, or to update an existing environment.

# Export an environment to a file
conda env export > environment.yml

# Create an environment from a file
conda env create -f environment.yml

# Update an environment from a file
conda env update -f environment.yml

Virtual Environments (venv/virtualenv)

Python provides venv as a buil-in tool for creating virtual environments. virtualenv is a third-party tool that provides similar functionality.

Basic commands

# Creating a virtual environment
# Using venv (Python 3.3+ built-in)
python -m venv your-env-name

# Using virtualenv (must be installed first)
pip install virtualenv
virtualenv your-env-name

#Activating the environment
# Linux/macOS
source your-env-name/bin/activate
# Windows
your-env-name\Scripts\activate

# Installing a library (package)
pip install lib_name

# Uninstalling a library (package)
pip uninstall lib_name

# To deactivate
deactivate

Managing dependencies with pip

A requirements.txt file lists all dependencies with their specific versions.

# Export requirements.txt from an activated environment
pip freeze > requirements.txt

# Install dependencies from requirements.txt
pip install -r requirements.txt
Tip

Use pip-chill or pipreqs instead of pip freeze to exclude unnecessary dependencies. pip-chill lists only packages you installed, while pipreqs lists packages your code actually uses.

Dependency Management Tools

Consider using tools that offer more sophisticated dependency management by integrating virtual environment creation and dependency resolution. They maintain a project manifest (e.g., pyproject.toml for Poetry) that specifies primary dependencies and generate lock files to pin exact versions for reproducibility.

  • Pipenv: Combines pip and virtualenv into a single tool, with a focus on simplicity and ease of use.
  • Poetry: Manages dependencies, environments, and package building in a streamlined way.
  • Pixi: A new tool that aims to provide a more user-friendly experience for managing Python environments and dependencies.
Learn more