Posted in

Surprisingly necessary explanation of Python venv

There is a strange phenomenon in the Python world. People train neural networks, automate infrastructure, write complex tools and maintain applications for years while still having very little understanding of Python virtual environments.

I’m not entirely sure why this happens. Maybe it’s because Python has such a low barrier to entry that beginners skip this topic just to get their first script running and then never come back to it as they progress further, focusing on Python syntax and programming skills instead.

At later stages of learning, this leads people to ask questions like:

  • “How do I back up all dependencies of my Python project?”
  • “How do I run pip install on a machine without internet access?”
  • “How do I know which dependencies my application actually uses?”

All of these problems can be solved using basic virtual environment features, but many developers never develop that intuition.

What venv is NOT

A Python virtual environment is not:

  • a container – because it will not isolate your python process from the rest of the system
  • a virtual machine – because it will not emulate computer components
  • a true sandbox – because it will still be able to access the filesystem, network requests etc. (although some people can argue that Python-level isolation is enough to call it a sandbox for the Python software development use case)

A venv is simply a directory where your project-specific interpreter’s configuration and project-specific dependencies live. Such an approach gives Python developers many useful features.

Dependency isolation (the one everybody usually knows)

Let’s start with the obvious one. Different projects require different dependencies. Sometimes the same dependencies, but in different versions. Installing everything globally in the system quickly creates chaos because some projects may start working after the dependency update and the others stop working due to the same reason.

Virtual environments solve this by giving each project its own package space:

MDX
project_a/.venv
project_b/.venv

Each environment contains independent:

  • installed packages
  • dependency versions
  • console scripts
  • package metadata

This shifts the problem from “works on my machine” to “works with this venv” which limits debugging only to the content of the used virtual environment.

No need to explicitly activate/deactivate the environment

When you run:

ShellScript
source .venv/bin/activate

the script mainly modifies your shell’s PATH variable so that .venv/bin appears before system directories. As a result, commands like python now resolve to executables inside the virtual environment.

Such activation is optional and when creating automations or running scripts remotely, it is actually a good idea to explicitly run the interpreter directly:

ShellScript
./.venv/bin/python script.py

or:

ShellScript
./.venv/bin/python -m pip install requests

AI is powerful. Snippets are instant.

Stop prompting for the same patterns repeatedly. Get almost 100 free VS Code snippets for C++, Python, CMake and Bazel from piko::snippets GitHub repository.

Build env on one machine and run on another

This is where things start becoming genuinely interesting. You may have already encountered the following setup:

Machine A:

  • you have full permissions
  • you can install packages with pip
  • has internet access

Machine B:

  • locked-down server
  • restricted permissions
  • no internet access

Many people assume this means that their Python tooling cannot be used on Machine B because how would they ensure proper dependencies. The answer is that you can build your dependencies as .whl packages, transfer them via internal network and install them offline on the server side. Let’s say you need empy for your application to work. On machine A create the virtual environment:

ShellScript
python3 -m venv machine_a

And then install empty with pip:

ShellScript
./machine_a/bin/python -m pip install empy

After it’s done, display the installed packages to make sure it’s there:

ShellScript
./machine_a/bin/python -m pip freeze
empy==4.2.1

Now, build a wheel package:

ShellScript
/machine_a/bin/python -m pip wheel empy==4.2.1 -w deps

This creates a deps folder with a empy-4.2.1-py3-none-any.whl file inside. Transfer that folder to the machine B and then create there a new virtual environment:

ShellScript
python3 -m venv machine_b

You can now install the dependency fully offline out of the transferred .whl file:

ShellScript
./machine_b/bin/python -m pip install deps/empy-4.2.1-py3-none-any.whl

App delivery as a packaged virtual environment

Let’s restrict the environment of the machine B from the previous paragraph even more and let’s say that you can’t even install dependencies with pip out of .whl packages. What can you do? You can transfer entire virtual environment folder from machine A to the machine B and run your script there.

The venv package will provide everything that your scripts need (of course, as long as both machines are sufficiently compatible – they share OS family, architecture etc.).

A dependency backup

Virtual environment folder captures your dependencies and their versions, so whenever you need to know what dependencies you actually use and in what versions, you can simply run:

ShellScript
./venv/bin/python -m pip freeze

If you save the output to the file (usually called requirements.txt) you can always re-create your venv by invoking:

ShellScript
./venv/bin/python -m pip install -r requirements.txt

It means that you can just copy your current venv folder, tweak versions of some dependencies, check if your app still works and if yes – start using these new versions. If no, you simply delete the new venv folder and use the old one.

Read also: