Multiple Version Testing

If you are interested in running multiple versions of the same development language using Nix and Nix Flakes, ride along while I setup a development environment that includes Python 2.7, 3.5-3.12 along with necessary development libraries. I will not be covering installing Nix on your system or the basics of the Nix package system or language. At the link above, you can see and fetch the flake.nix file. Each commit in the repository corresponds to a version of the file in this post.

If you only want to see the full flake file and get to using it for yourself, check down at the bottom of the post.

Caveat: This post assumes you have basic knowledge of Nix, its installation, and its file syntax.

Initial Flake Setup

Create a new folder and create your flake.nix file with the basic inputs. Tell it to install the Python 3 package and a shell to use.

{
    description = "A Python development environment";
    inputs = {
        nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
        flake-utils.url = "github:numtide/flake-utils";
    };

    outputs = {
        self,
        flake-utils,
        nixpkgs
    }: flake-utils.lib.eachDefaultSystem (system: let
        pkgs = nixpkgs.legacyPackages.${system};
    in {
        devShell = pkgs.mkShell {
            nativeBuildInputs = [ pkgs.bashInteractive ];
            buildInputs = [
                pkgs.python3
            ];
        };
    });
}

Enter the development environment with the command nix develop. This will put you into a shell with the Nix packages defined in the flake.nix file. Of course, if you add flake.nix and the generated flake.lock files to your repository, you can be sure that everyone else who works on a project is able to use the exact same set of packages across their systems.

Personally, I use direnv tied into my shell to automatically activate my devshells, but for the purposes of this exercise I am going to use the nix develop command to build and test the environments. Setting up direnv and distributing Nix flakes is beyond the scope of this post

With the current version of Nix unstable, I get the following output:

[greg@jude:~/tmp]$ python3 --version
Python 3.10.11

Add Python Packages

When developing with Python, it is common to use a number of Python packages. Nixpkgs comes with a huge number of them already packaged and it is easy to package more for it. For the purpose of demonstration, I will only add one packge to my installation, but you can add any Python packages that you need. To do this, we are going to call the withPackages function on the Python package and give it a function that returns the packages we want added. That function will look like this

        packages = python-packages: with python-packages; [
            requests
        ];

Similarly we invoke the function by changing

-pkgs.python3
+(pkgs.python3.withPackages packages)

When that is added to the Flake file, the whole file looks like this

{
    description = "A Python development environment";
    inputs = {
        nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
        flake-utils.url = "github:numtide/flake-utils";
    };

    outputs = {
        self,
        flake-utils,
        nixpkgs
    }: flake-utils.lib.eachDefaultSystem (system: let
        pkgs = nixpkgs.legacyPackages.${system};
        packages = python-packages: with python-packages; [
            requests
        ];
    in {
        devShell = pkgs.mkShell {
            nativeBuildInputs = [ pkgs.bashInteractive ];
            buildInputs = [
                (pkgs.python3.withPackages packages)
            ];
        };
    });
}

Now I can run nix develop again and get a shell where I can import the requests library in my Python interpreter.

Add more versions

Having one version of Python is good, but the point is to enable working with different versions. To do this, we will create a Flake input for each version from Python 3.5 through 3.11 that we have available. Creating a separate input, even when several of them will point to the same nixpkgs git branch, is necessary because each version will leave support at a different time. We want to be able to update the flake in the future and easily leave unsupported versions of Python on their latest Nix branch while also not holding back supported versions from their future patch releases. We also add a pin for Python 2.7, which is still in use in some legacy servers.

So let’s add a bunch of inputs like so:

nixpkgs311.url = "github:NixOS/nixpkgs/nixos-unstable";
nixpkgs310.url = "github:NixOS/nixpkgs/nixos-22.11";
nixpkgs39.url = "github:NixOS/nixpkgs/nixos-22.11";
nixpkgs38.url = "github:NixOS/nixpkgs/nixos-22.11";
nixpkgs37.url = "github:NixOS/nixpkgs/nixos-22.05";
nixpkgs36.url = "github:NixOS/nixpkgs/nixos-21.05";
nixpkgs35.url = "github:NixOS/nixpkgs/nixos-20.03";
nixpkgs27.url = "github:NixOS/nixpkgs/nixos-20.09";

At the time of writing NixOS 22.11 is the latest release and Python 3.8-3.11 are all still in support. Therefore, all of them are available in the 22.11 branch for the time being. At the same time we do this, we add the new inputs to the output function, update the identification of system support, and add the package to the available list.

Full Array

Now our flake looks like this:

{
    description = "A Python development environment";
    inputs = {
        nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
        flake-utils.url = "github:numtide/flake-utils";
        nixpkgs311.url = "github:NixOS/nixpkgs/nixos-unstable";
        nixpkgs310.url = "github:NixOS/nixpkgs/nixos-22.11";
        nixpkgs39.url = "github:NixOS/nixpkgs/nixos-22.11";
        nixpkgs38.url = "github:NixOS/nixpkgs/nixos-22.11";
        nixpkgs37.url = "github:NixOS/nixpkgs/nixos-22.05";
        nixpkgs36.url = "github:NixOS/nixpkgs/nixos-21.05";
        nixpkgs35.url = "github:NixOS/nixpkgs/nixos-20.03";
        nixpkgs27.url = "github:NixOS/nixpkgs/nixos-20.09";
    };

    outputs = {
        self,
        flake-utils,
        nixpkgs,
        nixpkgs311,
        nixpkgs310,
        nixpkgs39,
        nixpkgs38,
        nixpkgs37,
        nixpkgs36,
        nixpkgs35,
        nixpkgs27,
    }: flake-utils.lib.eachDefaultSystem (system: let
        pkgs = nixpkgs.legacyPackages.${system};
        pkgs311 = nixpkgs311.legacyPackages.${system};
        pkgs310 = nixpkgs310.legacyPackages.${system};
        pkgs39 = nixpkgs39.legacyPackages.${system};
        pkgs38 = nixpkgs38.legacyPackages.${system};
        pkgs37 = nixpkgs37.legacyPackages.${system};
        pkgs36 = nixpkgs36.legacyPackages.${system};
        pkgs35 = nixpkgs35.legacyPackages.${system};
        pkgs27 = nixpkgs27.legacyPackages.${system};
        packages = python-packages: with python-packages; [
            requests
        ];
    in {
        devShell = pkgs.mkShell {
            nativeBuildInputs = [ pkgs.bashInteractive ];
            buildInputs = [
                (pkgs.python3.withPackages packages)
                (pkgs311.python311.withPackages packages)
                (pkgs310.python310.withPackages packages)
                (pkgs39.python39.withPackages packages)
                (pkgs38.python38.withPackages packages)
                (pkgs37.python37.withPackages packages)
                (pkgs36.python36.withPackages packages)
                (pkgs35.python35.withPackages packages)
                (pkgs27.python27.withPackages packages)
            ];
        };
    });
}

We can test that they are all present by doing a nix develop. This build might take a long time the first time you run it on your machine, because older packages are not available in the binary cache for NixOS. Fortunately, all Nix packages include the entire history of how to build them from source, so you CAN get it done. It just might take a while. Let’s run a tiny Bash script to check that we have all the Python versions and that they all have the Requests library available:

for ver in $(seq 5 11); do
    python3.${ver} -c 'import sys, requests; print(sys.version_info)'
done

Sample output:

$ for ver in $(seq 5 11); do python3.${ver} -c 'import requests, sys; print(sys.version_info)'; done
sys.version_info(major=3, minor=5, micro=9, releaselevel='final', serial=0)
sys.version_info(major=3, minor=6, micro=14, releaselevel='final', serial=0)
sys.version_info(major=3, minor=7, micro=16, releaselevel='final', serial=0)
sys.version_info(major=3, minor=8, micro=16, releaselevel='final', serial=0)
sys.version_info(major=3, minor=9, micro=16, releaselevel='final', serial=0)
sys.version_info(major=3, minor=10, micro=11, releaselevel='final', serial=0)
sys.version_info(major=3, minor=11, micro=3, releaselevel='final', serial=0)

Likewise we can validate that Python 2.7 is available:

$ python2.7 -c 'import requests, sys; print(sys.version_info)'
sys.version_info(major=2, minor=7, micro=18, releaselevel='final', serial=0)

Python 3.0 to 3.4

These versions of Python were released before Nix was its current, modern self. There is not support in those Nixpkgs releases for Flakes which makes adding them to the environment a pain. I tried testing importing the tarball directly, but the actual builds of the old versions were broken. Versions 3.2-3.4 are in the repository but they do not build without major changes which would be beyond the scope of this post. Meanwhile Python 3.0 and 3.1 were released before Nix packages even existed and were never packaged for the main repository.