Silicon Soul

A wonderful part of using a GNU/Linux system is that it comes bundled with a powerful command-line interface (CLI) that not only helps you maintain your system, but introduces you to scripting and automation. Over time, you may amass a collection of personal scripts to meet your computing needs and find that they greatly enhance the quality of your digital life.

As your scripting abilities advance and you have more complicated demands, you may find that Bash no longer seems to be the right tool for the job. Perhaps your scripts are too long and you cannot find a sustainable way to maintain them. Maybe you have more complicated debugging needs, and the tools that the Bash shell offers are not sufficient.

On the other hand, maybe your scripting experiences have sparked an interest in object-oriented programming, and you feel the need to explore a full-featured programming language. If any of these possibilities resonate with you, learning Python may be a logical next step in your GNU/Linux journey.

Note: If you are not familiar with the GNU/Linux command line interface, review the Conventions page before proceeding.

No Installation Required

One advantage of using a GNU/Linux distribution like Debian or Fedora is that package maintainers have already done the work of selecting a version of Python for inclusion into each distribution. You do not have to determine where to obtain Python from or how to install it on your system. It is already there.

The exact version of Python installed on your system and how it is invoked differs depending on the GNU/Linux distribution that you are using. You can determine this with some CLI exploration.

For example, on Debian 11, running which python will not yield a file path. If we run find '/usr/bin' -iname '*python*', we would see something like the following

/usr/bin/python3
/usr/bin/python3.9

With a little more research, we can see that Debian 11 makes Python available through python3, and that python3 is a symbolic link that points to an actual version of a Python interpreter (specifically, python3.9):

$ ls -l '/usr/bin/python3'
lrwxrwxrwx 1 root root 9 Apr  5  2021 /usr/bin/python3 -> python3.9
$ file '/usr/bin/python3.9'
/usr/bin/python3.9: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=f902f8a561c3abdb9c8d8c859d4243bd8c3f928f, for GNU/Linux 3.2.0, stripped

On Fedora 35, things are a little more complicated:

$ find '/usr/bin' -iname '*python*'
/usr/bin/abrt-action-analyze-python
/usr/bin/activate-global-python-argcomplete
/usr/bin/python-argcomplete-check-easy-install-script
/usr/bin/python-argcomplete-tcsh
/usr/bin/register-python-argcomplete
/usr/bin/reporter-bugzilla-python
/usr/bin/python
/usr/bin/python3
/usr/bin/python3.10

Above, we can see that Fedora 35 makes Python available through both python and python3. python is a symbolic link to python3 and, like on Debian 11, python3 is a symbolic link to an actual version of Python. This time, the actual Python interpreter version is 3.10:

$ ls -l '/usr/bin/python' '/usr/bin/python3'
lrwxrwxrwx. 1 root root  9 Nov 14 23:23 /usr/bin/python -> ./python3
lrwxrwxrwx. 1 root root 10 Nov 14 17:51 /usr/bin/python3 -> python3.10
$ file '/usr/bin/python3.10'
/usr/bin/python3.10: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=96e8ddd82f3b8790a0a78c3a1e99a5f998bfa345, for GNU/Linux 3.2.0, stripped

To summarize, when you are ready to use Python on either Debian or Fedora, it will be available via python3.

pip

One benefit of using Python is that it has a large assortment of first- and third-party software available for you to use. First-party software is included with your version of Python via the standard library. Third-party software can be primarily found via the Python Package Index (PyPI).

First-party software from the standard library can be pulled into a Python session or program with an import statement. However, a stream-lined solution is needed for obtaining and maintaining third-party software. pip (pip installs packages) is the recommended solution.

pip is the package installer for Python. It can work with any Python index, but is configured to work with PyPI by default.

Unlike Python itself, pip does not come with Debian or Fedora. It needs to be installed:

# apt install python3-pip (Debian)

# dnf install python3-pip (Fedora)

pip is both a script and a module. In Python, these and related terms have special meanings:

Scripts

Top-level files intended for execution by the user.

Modules

Collections of related code intended to be imported. Modules extend the power of scripts.

Packages

A directory of modules. Packages allow the hierarchical structure of the module namespace. Like how files are organized on a drive into folders and sub-folders, modules are organized into packages and subpackages.

To be considered a package (or subpackage), a directory must contain a file named __init__.py. Usually, this file includes the initialization code for the corresponding package.

Note that a package is still a module.

Libraries

An umbrella term referring to a reusable collection of code. Usually, a Python library is a directory of related modules and packages.

After installing pip, you will likely see multiple versions of it installed. Instead of selecting a version to run, you are going to select your Python interpreter (i.e., python3), and then tell it that you want to execute pip for your chosen interpreter.

Specifically, we tell our Python interpreter that we want to run the pip module as a script using python3's -m option:

python3 -m pip

Running the command above should show you pip's usage information, as well as its general options and subcommands. pip (or pip subcommand) help information can also be viewed by using its -h (--help) option, e.g., python3 -m pip -h, python3 -m pip install -h.

Virtual Environments

The system version of Python that comes with your GNU/Linux distribution is maintained by your distribution, and can be managed via your distribution's package manager. However, things can get more complicated when managing Python packages via pip.

For example, there may be third-party software that you only need for a specific project. Another common issue is that you need to be able to share a project with other people. It would be nice if there was a way for them to recreate your Python environment with the same third-party software at the same version numbers, ensuring that they can run your project just like you would.

A Python virtual environment offers a solution for these scenarios. Virtual environments create an isolated environment without disrupting other Python applications running on your system. They are self-contained directory trees that contain a Python installation for a particular version of Python (e.g., the version of Python packaged with your GNU/Linux distribution).

When you need to install software via pip, it is often best to do so inside of a virtual environment.

Virtual environments can be created with the standard library's venv module. For Debian, the python3-venv package should be installed before proceeding:

# apt install python3-venv

If you are using Fedora, you are already good to go.

Creating Virtual Environments

Storing virtual environments in a single directory (e.g., ${HOME}/venvs/) is common. So, for example, we can create a virtual environment for building static sites in ${HOME}/venvs/ with the following command (if ${HOME}/venvs/ does not exist, the command will create it for us):

python3 -m venv --upgrade-deps "${HOME}/venvs/static_sites/"

The --upgrade-deps venv option upgrades the virtual environment's versions of pip and setuptools after the environment's creation.

Working With Virtual Environments

If we change to ${HOME}/venvs/static_sites/ and list out its contents, we will see the root of the self-contained directory tree that venv created for us:

$ cd "${HOME}/venvs/static_sites/" && ls -l
total 20
drwxr-xr-x 2 amnesia amnesia 4096 Jan 21 14:36 bin
drwxr-xr-x 2 amnesia amnesia 4096 Jan 21 14:35 include
drwxr-xr-x 3 amnesia amnesia 4096 Jan 21 14:35 lib
lrwxrwxrwx 1 amnesia amnesia    3 Jan 21 14:35 lib64 -> lib
-rw-r--r-- 1 amnesia amnesia   69 Jan 21 14:35 pyvenv.cfg
drwxr-xr-x 3 amnesia amnesia 4096 Jan 21 14:35 share

The tree command (# apt install tree for Debian) can help us visualize the beginnings of the tree and understand what is going on:

$ tree -L 2
.
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── Activate.ps1
│   ├── pip
│   ├── pip3
│   ├── pip3.10
│   ├── pip3.9
│   ├── python -> python3
│   ├── python3 -> /usr/bin/python3
│   └── python3.9 -> python3
├── include
├── lib
│   └── python3.9
├── lib64 -> lib
├── pyvenv.cfg
└── share
    └── python-wheels


7 directories, 12 files

We can see that for this Debian 11 installation, python and python3.9 are symbolic links that point to python3, and that python3 is also a symbolic link that points to /usr/bin/python3. This demonstrates how a venv virtual environment is tied to our distribution's version of Python.

This is important to keep in mind. If your distribution's version of Python changes (e.g., when you upgrade your distribution to the next major version, which likely upgrades your version of Python), your virtual environments will no longer work. Fortunately, you can upgrade your virtual environments to use the current version of Python on your GNU/Linux distribution:

python3 -m venv --upgrade --upgrade-deps ex_ve_dir...

The pyvenv.cfg file contains the virtual environment's configuration:

$ cat pyvenv.cfg
home = /usr/bin
include-system-site-packages = false
version = 3.9.2

• home represents the Python installation directory from which the python3 -m venv command was run to create the virtual environment.
• include-system-site-packages determines whether the global site-packages location should be included (defaults to false, i.e., the virtual environment is isolated from the system's Python environment).
• version confirms the virtual environment's Python version.

The lib/ directory is also important. This directory contains the pythonX.X/site-packages subdirectory where software installed to the virtual environment is stored. For example, we can see pip and setuptools here, which were upgraded after virtual environment creation:

$ ls -l 'lib/python3.9/site-packages/'
total 32
drwxr-xr-x 3 amnesia amnesia 4096 Jan 21 14:36 _distutils_hack
-rw-r--r-- 1 amnesia amnesia  151 Jan 21 14:36 distutils-precedence.pth
drwxr-xr-x 5 amnesia amnesia 4096 Jan 21 14:36 pip
drwxr-xr-x 2 amnesia amnesia 4096 Jan 21 14:36 pip-22.3.1.dist-info
drwxr-xr-x 5 amnesia amnesia 4096 Jan 21 14:36 pkg_resources
drwxr-xr-x 2 amnesia amnesia 4096 Jan 21 14:35 pkg_resources-0.0.0.dist-info
drwxr-xr-x 8 amnesia amnesia 4096 Jan 21 14:36 setuptools
drwxr-xr-x 2 amnesia amnesia 4096 Jan 21 14:36 setuptools-66.1.1.dist-info

Finally, the bin/activate shell script should be noted. Before using a virtual environment, the environment can be activated.

Activating the virtual environment is not technically required to use it, but activation does several important things, including:

• Changing the shell’s prompt to show the virtual environment that you are using.
• Modifying the environment so that running python gets you the specific version and installation of Python that you used to create the virtual environment with (e.g., ${HOME}/venvs/static_sites/bin/python -> ${HOME}/venvs/static_sites/bin/python3 -> /usr/bin/python3).

• Prepending the virtual environment's bin/ directory to the PATH environment variable.

  This ensures that software you install in the virtual environment is given precedence in your shell's environment and enables you to just use the name of the software to invoke it, versus using its path.

• Setting the VIRTUAL_ENV environment variable to your virtual environment's directory.

• Registering the deactivate shell function, which undoes the above steps.

Activation can be accomplished by loading the virtual environment's bin/activate file into the current shell environment with the source command. For example, if your current directory is ${HOME}/venvs/static_sites/, you could run the following command to activate the static_sites virtual environment:

source bin/activate

Afterwards, your shell's command prompt should be updated with the virtual environment's name:

(static_sites) $

Now, you are ready to begin installing software into your virtual environment.

python3 -m pip install ex_pkg...

For example, the following command installs the wheel package, which is handy for building and working with Python wheel files. After creating a virtual environment, you may want to have it installed:

python3 -m pip install wheel

Since we are working with static sites for our example, we install pelican, a Python-powered static site generator, too:

python3 -m pip install pelican

A virtual environment can be deactivated from any file system location by entering deactivate, or by exiting the shell from which you activated the virtual environment.

Installed Packages

pip provides a great way to both record installed Python packages and to quickly re-install them. When combined with a virtual environment, this enables us to easily recreate virtual environments with the exact software versions installed when the virtual environment installed packages were first recorded.

We can save the installed Python packages to a specially formatted requirements.txt file using pip freeze:

python3 -m pip freeze > requirements.txt

python3 -m pip freeze | grep -v 'pkg_resources' > requirements.txt (Debian)

For example, if we wanted to do this for our ${HOME}/venvs/static_sites/ virtual environment, we would ensure that the environment was first activated and then run the following:

(static_sites) $ python3 -m pip freeze |
    grep -v 'pkg_resources' > requirements.txt

If we examined the requirements.txt file, it might look something like this:

blinker==1.6
docutils==0.19
feedgenerator==2.0.0
Jinja2==3.1.2
markdown-it-py==2.2.0
MarkupSafe==2.1.2
mdurl==0.1.2
pelican==4.8.0
Pygments==2.14.0
python-dateutil==2.8.2
pytz==2023.3
rich==13.3.3
six==1.16.0
typing_extensions==4.5.0
Unidecode==1.3.6

If we or a collaborator need to recreate this project's virtual environment, we can do so with this requirements.txt file like so:

python3 -m pip install -r 'requirements.txt'

For example, if a collaborator downloaded our requirements.txt file to ${HOME}/Downloads/ and had a new ${HOME}/venvs/test_venv/ virtual environment they wanted to recreate our static_sites virtual environment in, they could run the following command after activating the test_venv virtual environment:

(test_venv) $ python3 -m pip install -r "${HOME}/Downloads/requirements.txt"

Data/Content Recommendations

A virtual environment may need to be something that is labile and/or ephemeral, so it may not be best to store your project data inside of it. However, the virtual environment software you install needs a way to access this data.

To achieve this, it may be best to store your project data outside of its virtual environment and to symbolically link to it from within the virtual environment. For example, we can store content for a static site at ${HOME}/sites/pelican_proj/ and then create a symbolic link to it inside of our ${HOME}/venvs/static_sites/ virtual environment:

ln -s ${HOME}/sites/pelican_proj/ ${HOME}/venvs/static_sites/pelican_proj

Now, if the static_sites virtual environment ever needs to be recreated, you just need to recreate the symbolic link for it to regain access to its data.

A Solid Foundation

Learning how to set up a development environment is often one of the most difficult and underappreciated parts of getting started with a new technology. There may not be a universally accepted way of developing with Python, but there are many commonly accepted best practices that can help you start out on solid ground.