A wonderful part of using a GNU/Linux system is that it comes bundled with a powerful command-line interface (CLI) that not only helps you maintain your system, but introduces you to scripting and automation. Over time, you may amass a collection of personal scripts to meet your computing needs and find that they greatly enhance the quality of your digital life.
As your scripting abilities advance and you have more complicated demands, you may find that Bash no longer seems to be the right tool for the job. Perhaps your scripts are too long and you cannot find a sustainable way to maintain them. Maybe you have more complicated debugging needs, and the tools that the Bash shell offers are not sufficient.
On the other hand, maybe your scripting experiences have sparked an interest in object-oriented programming, and you feel the need to explore a full-featured programming language. If any of these possibilities resonate with you, learning Python may be a logical next step in your GNU/Linux journey.
Note: If you are not familiar with the GNU/Linux command line interface, review the Conventions page before proceeding.
No Installation Required
One advantage of using a GNU/Linux distribution like Debian or Fedora is that package maintainers have already done the work of selecting a version of Python for inclusion into each distribution. You do not have to determine where to obtain Python from or how to install it on your system. It is already there.
The exact version of Python installed on your system and how it is invoked differs depending on the GNU/Linux distribution that you are using. You can determine this with some CLI exploration.
For example, on Debian 11, running which python
will not yield a file path. If we run find '/usr/bin' -iname '*python*'
, we would see something like the following
/usr/bin/python3
/usr/bin/python3.9
With a little more research, we can see that Debian 11 makes Python available through python3
, and that python3
is a symbolic link that points to an actual version of a Python interpreter (specifically, python3.9
):
$ ls -l '/usr/bin/python3'
lrwxrwxrwx 1 root root 9 Apr 5 2021 /usr/bin/python3 -> python3.9
$ file '/usr/bin/python3.9'
/usr/bin/python3.9: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=f902f8a561c3abdb9c8d8c859d4243bd8c3f928f, for GNU/Linux 3.2.0, stripped
On Fedora 35, things are a little more complicated:
$ find '/usr/bin' -iname '*python*'
/usr/bin/abrt-action-analyze-python
/usr/bin/activate-global-python-argcomplete
/usr/bin/python-argcomplete-check-easy-install-script
/usr/bin/python-argcomplete-tcsh
/usr/bin/register-python-argcomplete
/usr/bin/reporter-bugzilla-python
/usr/bin/python
/usr/bin/python3
/usr/bin/python3.10
Above, we can see that Fedora 35 makes Python available through both python
and python3
. python
is a symbolic link to python3
and, like on Debian 11, python3
is a symbolic link to an actual version of Python. This time, the actual Python interpreter version is 3.10
:
$ ls -l '/usr/bin/python' '/usr/bin/python3'
lrwxrwxrwx. 1 root root 9 Nov 14 23:23 /usr/bin/python -> ./python3
lrwxrwxrwx. 1 root root 10 Nov 14 17:51 /usr/bin/python3 -> python3.10
$ file '/usr/bin/python3.10'
/usr/bin/python3.10: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=96e8ddd82f3b8790a0a78c3a1e99a5f998bfa345, for GNU/Linux 3.2.0, stripped
To summarize, when you are ready to use Python on either Debian or Fedora, it will be available via python3
.
pip
One benefit of using Python is that it has a large assortment of first- and third-party software available for you to use. First-party software is included with your version of Python via the standard library. Third-party software can be primarily found via the Python Package Index (PyPI).
First-party software from the standard library can be pulled into a Python session or program with an import
statement. However, a stream-lined solution is needed for obtaining and maintaining third-party software. pip (pip installs packages) is the recommended solution.
pip is the package installer for Python. It can work with any Python index, but is configured to work with PyPI by default.
Unlike Python itself, pip does not come with Debian or Fedora. It needs to be installed:
# apt install python3-pip
(Debian)
# dnf install python3-pip
(Fedora)
pip
is both a script and a module. In Python, these and related terms have special meanings:
- Scripts
- Top-level files intended for execution by the user.
- Modules
- Collections of related code intended to be imported. Modules extend the power of scripts.
- Packages
- A directory of modules. Packages allow the hierarchical structure of the module namespace. Like how files are organized on a drive into folders and sub-folders, modules are organized into packages and subpackages.
- To be considered a package (or subpackage), a directory must contain a file named
__init__.py
. Usually, this file includes the initialization code for the corresponding package. - Note that a package is still a module.
- Libraries
- An umbrella term referring to a reusable collection of code. Usually, a Python library is a directory of related modules and packages.
After installing pip
, you will likely see multiple versions of it installed. Instead of selecting a version to run, you are going to select your Python interpreter (i.e., python3
), and then tell it that you want to execute pip
for your chosen interpreter.
Specifically, we tell our Python interpreter that we want to run the pip
module as a script using python3
's -m
option:
python3 -m pip
Running the command above should show you pip
's usage information, as well as its general options and subcommands. pip
(or pip
subcommand) help information can also be viewed by using its -h
(--help
) option, e.g., python3 -m pip -h
, python3 -m pip install -h
.
Virtual Environments
The system version of Python that comes with your GNU/Linux distribution is maintained by your distribution, and can be managed via your distribution's package manager. However, things can get more complicated when managing Python packages via pip
.
For example, there may be third-party software that you only need for a specific project. Another common issue is that you need to be able to share a project with other people. It would be nice if there was a way for them to recreate your Python environment with the same third-party software at the same version numbers, ensuring that they can run your project just like you would.
A Python virtual environment offers a solution for these scenarios. Virtual environments create an isolated environment without disrupting other Python applications running on your system. They are self-contained directory trees that contain a Python installation for a particular version of Python (e.g., the version of Python packaged with your GNU/Linux distribution).
When you need to install software via pip
, it is often best to do so inside of a virtual environment.
Virtual environments can be created with the standard library's venv
module. For Debian, the python3-venv
package should be installed before proceeding:
# apt install python3-venv
If you are using Fedora, you are already good to go.
Creating Virtual Environments
Storing virtual environments in a single directory (e.g., ${HOME}/venvs/
) is common. So, for example, we can create a virtual environment for building static sites in ${HOME}/venvs/
with the following command (if ${HOME}/venvs/
does not exist, the command will create it for us):
python3 -m venv --upgrade-deps "${HOME}/venvs/static_sites/"
The --upgrade-deps
venv
option upgrades the virtual environment's versions of pip
and setuptools
after the environment's creation.
Working With Virtual Environments
If we change to ${HOME}/venvs/static_sites/
and list out its contents, we will see the root of the self-contained directory tree that venv
created for us:
$ cd "${HOME}/venvs/static_sites/" && ls -l
total 20
drwxr-xr-x 2 amnesia amnesia 4096 Jan 21 14:36 bin
drwxr-xr-x 2 amnesia amnesia 4096 Jan 21 14:35 include
drwxr-xr-x 3 amnesia amnesia 4096 Jan 21 14:35 lib
lrwxrwxrwx 1 amnesia amnesia 3 Jan 21 14:35 lib64 -> lib
-rw-r--r-- 1 amnesia amnesia 69 Jan 21 14:35 pyvenv.cfg
drwxr-xr-x 3 amnesia amnesia 4096 Jan 21 14:35 share
The tree
command (# apt install tree
for Debian) can help us visualize the beginnings of the tree and understand what is going on:
$ tree -L 2
.
├── bin
│ ├── activate
│ ├── activate.csh
│ ├── activate.fish
│ ├── Activate.ps1
│ ├── pip
│ ├── pip3
│ ├── pip3.10
│ ├── pip3.9
│ ├── python -> python3
│ ├── python3 -> /usr/bin/python3
│ └── python3.9 -> python3
├── include
├── lib
│ └── python3.9
├── lib64 -> lib
├── pyvenv.cfg
└── share
└── python-wheels
7 directories, 12 files
We can see that for this Debian 11 installation, python
and python3.9
are symbolic links that point to python3
, and that python3
is also a symbolic link that points to /usr/bin/python3
. This demonstrates how a venv
virtual environment is tied to our distribution's version of Python.
This is important to keep in mind. If your distribution's version of Python changes (e.g., when you upgrade your distribution to the next major version, which likely upgrades your version of Python), your virtual environments will no longer work. Fortunately, you can upgrade your virtual environments to use the current version of Python on your GNU/Linux distribution:
python3 -m venv --upgrade --upgrade-deps ex_ve_dir...
The pyvenv.cfg
file contains the virtual environment's configuration:
$ cat pyvenv.cfg
home = /usr/bin
include-system-site-packages = false
version = 3.9.2
home
represents the Python installation directory from which thepython3 -m venv
command was run to create the virtual environment.include-system-site-packages
determines whether the globalsite-packages
location should be included (defaults tofalse
, i.e., the virtual environment is isolated from the system's Python environment).version
confirms the virtual environment's Python version.
The lib/
directory is also important. This directory contains the pythonX.X/site-packages
subdirectory where software installed to the virtual environment is stored. For example, we can see pip
and setuptools
here, which were upgraded after virtual environment creation:
$ ls -l 'lib/python3.9/site-packages/'
total 32
drwxr-xr-x 3 amnesia amnesia 4096 Jan 21 14:36 _distutils_hack
-rw-r--r-- 1 amnesia amnesia 151 Jan 21 14:36 distutils-precedence.pth
drwxr-xr-x 5 amnesia amnesia 4096 Jan 21 14:36 pip
drwxr-xr-x 2 amnesia amnesia 4096 Jan 21 14:36 pip-22.3.1.dist-info
drwxr-xr-x 5 amnesia amnesia 4096 Jan 21 14:36 pkg_resources
drwxr-xr-x 2 amnesia amnesia 4096 Jan 21 14:35 pkg_resources-0.0.0.dist-info
drwxr-xr-x 8 amnesia amnesia 4096 Jan 21 14:36 setuptools
drwxr-xr-x 2 amnesia amnesia 4096 Jan 21 14:36 setuptools-66.1.1.dist-info
Finally, the bin/activate
shell script should be noted. Before using a virtual environment, the environment can be activated.
Activating the virtual environment is not technically required to use it, but activation does several important things, including:
- Changing the shell’s prompt to show the virtual environment that you are using.
- Modifying the environment so that running
python
gets you the specific version and installation of Python that you used to create the virtual environment with (e.g.,${HOME}/venvs/static_sites/bin/python
->${HOME}/venvs/static_sites/bin/python3
->/usr/bin/python3
). -
Prepending the virtual environment's
bin/
directory to thePATH
environment variable.This ensures that software you install in the virtual environment is given precedence in your shell's environment and enables you to just use the name of the software to invoke it, versus using its path.
-
Setting the
VIRTUAL_ENV
environment variable to your virtual environment's directory. - Registering the
deactivate
shell function, which undoes the above steps.
Activation can be accomplished by loading the virtual environment's bin/activate
file into the current shell environment with the source
command. For example, if your current directory is ${HOME}/venvs/static_sites/
, you could run the following command to activate the static_sites
virtual environment:
source bin/activate
Afterwards, your shell's command prompt should be updated with the virtual environment's name:
(static_sites) $
Now, you are ready to begin installing software into your virtual environment.
python3 -m pip install ex_pkg...
For example, the following command installs the wheel
package, which is handy for building and working with Python wheel files. After creating a virtual environment, you may want to have it installed:
python3 -m pip install wheel
Since we are working with static sites for our example, we install pelican
, a Python-powered static site generator, too:
python3 -m pip install pelican
A virtual environment can be deactivated from any file system location by entering deactivate
, or by exiting the shell from which you activated the virtual environment.
Installed Packages
pip
provides a great way to both record installed Python packages and to quickly re-install them. When combined with a virtual environment, this enables us to easily recreate virtual environments with the exact software versions installed when the virtual environment installed packages were first recorded.
We can save the installed Python packages to a specially formatted requirements.txt
file using pip freeze
:
python3 -m pip freeze > requirements.txt
python3 -m pip freeze | grep -v 'pkg_resources' > requirements.txt
(Debian)
For example, if we wanted to do this for our ${HOME}/venvs/static_sites/
virtual environment, we would ensure that the environment was first activated and then run the following:
(static_sites) $ python3 -m pip freeze |
grep -v 'pkg_resources' > requirements.txt
If we examined the requirements.txt
file, it might look something like this:
blinker==1.6
docutils==0.19
feedgenerator==2.0.0
Jinja2==3.1.2
markdown-it-py==2.2.0
MarkupSafe==2.1.2
mdurl==0.1.2
pelican==4.8.0
Pygments==2.14.0
python-dateutil==2.8.2
pytz==2023.3
rich==13.3.3
six==1.16.0
typing_extensions==4.5.0
Unidecode==1.3.6
If we or a collaborator need to recreate this project's virtual environment, we can do so with this requirements.txt
file like so:
python3 -m pip install -r 'requirements.txt'
For example, if a collaborator downloaded our requirements.txt
file to ${HOME}/Downloads/
and had a new ${HOME}/venvs/test_venv/
virtual environment they wanted to recreate our static_sites
virtual environment in, they could run the following command after activating the test_venv
virtual environment:
(test_venv) $ python3 -m pip install -r "${HOME}/Downloads/requirements.txt"
Data/Content Recommendations
A virtual environment may need to be something that is labile and/or ephemeral, so it may not be best to store your project data inside of it. However, the virtual environment software you install needs a way to access this data.
To achieve this, it may be best to store your project data outside of its virtual environment and to symbolically link to it from within the virtual environment. For example, we can store content for a static site at ${HOME}/sites/pelican_proj/
and then create a symbolic link to it inside of our ${HOME}/venvs/static_sites/
virtual environment:
ln -s ${HOME}/sites/pelican_proj/ ${HOME}/venvs/static_sites/pelican_proj
Now, if the static_sites
virtual environment ever needs to be recreated, you just need to recreate the symbolic link for it to regain access to its data.
A Solid Foundation
Learning how to set up a development environment is often one of the most difficult and underappreciated parts of getting started with a new technology. There may not be a universally accepted way of developing with Python, but there are many commonly accepted best practices that can help you start out on solid ground.