Page Body

Getting Started With Python

A wonderful part of using a GNU/Linux system is that it comes bundled with a powerful command-line interface (CLI) that not only helps you maintain your system, but introduces you to scripting and automation. Over time, you may amass a collection of personal scripts to meet your computing needs and find that they greatly enhance the quality of your digital life.

As your scripting abilities advance and you have more complicated demands, you may find that Bash no longer seems to be the right tool for the job. Perhaps your scripts are too long and you cannot find a sustainable way to maintain them. Maybe you have more complicated debugging needs, and the tools that the Bash shell offers are not sufficient.

On the other hand, maybe your scripting experiences have sparked an interest in object-oriented programming, and you feel the need to explore a full-featured programming language. If any of these possibilities resonate with you, learning Python may be a logical next step in your GNU/Linux journey.

Note: If you are not familiar with the GNU/Linux command line interface, review the Conventions page before proceeding.

No Installation Required

One advantage of using a GNU/Linux distribution like Debian or Fedora is that package maintainers have already done the work of selecting a version of Python for inclusion into each distribution. You do not have to determine where to obtain Python from or how to install it on your system. It is already there.

The exact version of Python installed on your system and how it is invoked differs depending on the GNU/Linux distribution that you are using. You can determine this with some CLI exploration.

For example, on Debian 11, running which python will not yield a file path. If we run find '/usr/bin' -iname '*python*', we would see something like the following

/usr/bin/python3
/usr/bin/python3.9

With a little more research, we can see that Debian 11 makes Python available through python3, and that python3 is a symbolic link that points to an actual version of Python (specifically, python3.9):

$ ls -l '/usr/bin/python3'
lrwxrwxrwx 1 root root 9 Apr  5  2021 /usr/bin/python3 -> python3.9
$ file '/usr/bin/python3.9'
/usr/bin/python3.9: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=f902f8a561c3abdb9c8d8c859d4243bd8c3f928f, for GNU/Linux 3.2.0, stripped

On Fedora 35, things are a little more complicated:

$ find '/usr/bin' -iname '*python*'
/usr/bin/abrt-action-analyze-python
/usr/bin/activate-global-python-argcomplete
/usr/bin/python-argcomplete-check-easy-install-script
/usr/bin/python-argcomplete-tcsh
/usr/bin/register-python-argcomplete
/usr/bin/reporter-bugzilla-python
/usr/bin/python
/usr/bin/python3
/usr/bin/python3.10

Above, we can see that Fedora 35 makes Python available through both python and python3. python is a symbolic link to python3 and, like on Debian 11, python3 is a symbolic link to an actual version of Python. This time, the actual Python version is 3.10:

$ ls -l '/usr/bin/python' '/usr/bin/python3'
lrwxrwxrwx. 1 root root  9 Nov 14 23:23 /usr/bin/python -> ./python3
lrwxrwxrwx. 1 root root 10 Nov 14 17:51 /usr/bin/python3 -> python3.10
$ file '/usr/bin/python3.10'
/usr/bin/python3.10: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=96e8ddd82f3b8790a0a78c3a1e99a5f998bfa345, for GNU/Linux 3.2.0, stripped

To summarize, when you are ready to use Python on either Debian or Fedora, it will be available via python3.

pip

One benefit of using Python is that it has a large assortment of first- and third-party software available for you to use. First-party software is included with your version of Python via the standard library. Third-party software can be primarily found via the Python Package Index (PyPI).

First-party software from the standard library can be pulled into a Python session or program with an import statement. However, a stream-lined solution is needed for obtaining and maintaining third-party software. pip is the recommended solution.

pip is the package installer for Python. It can work with any Python index, but is configured to work with PyPI by default.

Unlike Python itself, pip does not come with Debian or Fedora. It needs to be installed:

# apt install python3-pip (Debian)

# dnf install python3-pip (Fedora)

pip is both a script and a module. In Python, these and related terms have special meanings:

Scripts
Top-level files intended for execution by the user.
Modules
Collections of related code intended to be imported. Modules extend the power of the scripts that you write.
Packages
A directory of modules. Packages allow the hierarchical structure of the module namespace. Just like we organize our files on a drive into folders and subfolders, we can organize our modules into packages and subpackages.
To be considered a package (or subpackage), a directory must contain a file named __init__.py. Usually, this file includes the initialization code for the corresponding package.
Note that a package is still a module.
Libraries
An umbrella term referring to a reusable collection of code. Usually, a Python library is a directory of related modules and packages.

After installing pip, you will likely see multiple versions of it installed. Instead of selecting a version to run, you are going to select your Python interpreter (i.e., python3), and then tell it that you want to execute pip for your chosen interpreter.

Specifically, we tell our Python interpreter that we want to run the pip module as a script using python3's -m option:

python3 -m pip

Running the command above should show you pip's usage information, as well as its general options and subcommands.

User Installs

The system version of Python that comes with your GNU/Linux distribution is maintained by your distribution, and can be managed via your distribution's package manager. However, things can get more complicated when managing Python packages via pip.

To simplify Python package installations, it may be best to utilize user installs. These installations use Python's user scheme for installation and provide an install location that is specific to your user.

User installs are accomplished by passing the --user option to the pip install subcommand:

python3 -m pip install --user ex_pkg...

For example, the following command can be used to install/upgrade your user's versions of pip, setuptools, and wheel:

python3 -m pip install -U --user pip setuptools wheel

The base directory for user installs is located at ${HOME}/.local/. You can confirm this location on your system by running python3 -m site --user-base.

If you have not already done so, add the base directory to your shell environment's PATH.

Virtual Environments

User installs are a great way to segment your personal Python software installs from the rest of your system. However, there may be times when even greater segmentation is required.

For example, there may be third-party software that you only need for a specific project. Another common issue is that you need to be able to share a project with other people. It would be nice if there was a way for them to recreate your Python environment with the same third-party software at the same version numbers, ensuring that they can run your project just like you would.

A Python virtual environment offers a solution for these scenarios. Virtual environments create an isolated environment without disrupting other Python applications running on your system. They are self-contained directory trees that contain a Python installation for a particular version of Python (e.g., the version of Python packaged with your GNU/Linux distribution).

Virtual environments can be created with the standard library's venv module. For Debian, the python3-venv package should be installed before proceeding:

# apt install python3-venv

If you are using Fedora, you are already good to go.

Creating Virtual Environments

Storing virtual environments in a single directory (e.g., ${HOME}/venvs/) is common. So, for example, we can create a virtual environment for building static sites in ${HOME}/venvs/ with the following command (if ${HOME}/venvs/ does not exist, the command will create it for us):

python3 -m venv --upgrade-deps "${HOME}/venvs/static_sites/"

The --upgrade-deps venv option upgrades the virtual environment's versions of pip and setuptools after the environment's creation.

Working With Virtual Environments

If we change to ${HOME}/venvs/static_sites/ and list out its contents, we will see the root of the self-contained directory tree that venv created for us:

$ cd "${HOME}/venvs/static_sites/" && ls -l
total 20
drwxr-xr-x 2 amnesia amnesia 4096 Jan 21 14:36 bin
drwxr-xr-x 2 amnesia amnesia 4096 Jan 21 14:35 include
drwxr-xr-x 3 amnesia amnesia 4096 Jan 21 14:35 lib
lrwxrwxrwx 1 amnesia amnesia    3 Jan 21 14:35 lib64 -> lib
-rw-r--r-- 1 amnesia amnesia   69 Jan 21 14:35 pyvenv.cfg
drwxr-xr-x 3 amnesia amnesia 4096 Jan 21 14:35 share

The tree command (# apt install tree for Debian) can help us visualize the beginnings of the tree and understand what is going on:

$ tree -L 2
.
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── Activate.ps1
│   ├── pip
│   ├── pip3
│   ├── pip3.10
│   ├── pip3.9
│   ├── python -> python3
│   ├── python3 -> /usr/bin/python3
│   └── python3.9 -> python3
├── include
├── lib
│   └── python3.9
├── lib64 -> lib
├── pyvenv.cfg
└── share
    └── python-wheels


7 directories, 12 files

We can see that for this Debian 11 installation, python and python3.9 are symbolic links that point to python3, and that python3 is also a symbolic link that points to /usr/bin/python3. This demonstrates how a venv virtual environment is tied to our distribution's version of Python.

This is important to keep in mind. If your distribution's version of Python changes (e.g., when you upgrade your distribution to the next major version, which likely upgrades your version of Python), your virtual environments will no longer work. Fortunately, you can upgrade your virtual environments to use the current version of Python on your GNU/Linux distribution:

python3 -m venv --upgrade ex_ve_dir... &&
    python3 -m venv --upgrade-deps ex_ve_dir...

The lib/ directory is also important. This directory contains the subdirectories where software installed to the virtual environment will be stored. For example, we can see pip and setuptools here, which were upgraded after virtual environment creation:

$ ls -l 'lib/python3.9/site-packages/'
total 32
drwxr-xr-x 3 amnesia amnesia 4096 Jan 21 14:36 _distutils_hack
-rw-r--r-- 1 amnesia amnesia  151 Jan 21 14:36 distutils-precedence.pth
drwxr-xr-x 5 amnesia amnesia 4096 Jan 21 14:36 pip
drwxr-xr-x 2 amnesia amnesia 4096 Jan 21 14:36 pip-22.3.1.dist-info
drwxr-xr-x 5 amnesia amnesia 4096 Jan 21 14:36 pkg_resources
drwxr-xr-x 2 amnesia amnesia 4096 Jan 21 14:35 pkg_resources-0.0.0.dist-info
drwxr-xr-x 8 amnesia amnesia 4096 Jan 21 14:36 setuptools
drwxr-xr-x 2 amnesia amnesia 4096 Jan 21 14:36 setuptools-66.1.1.dist-info

Finally, the bin/activate shell script should be noted. Before using a virtual environment, the environment needs to be activated. This can be accomplished by loading the virtual environment's bin/activate file into the current shell environment with the source command.

For example, if your current directory is ${HOME}/venvs/static_sites/, you could run the following command to activate the static_sites virtual environment:

source bin/activate

Afterwards, you should see the name of the virtual environment prepended to your shell's command prompt:

(static_sites) $

Now, you are ready to begin installing software into your virtual environment. By default, venv creates --no-site-packages virtual environments because the user site is not on the Python path. Therefore, when installing software in the virtual environment, pip install's --user option can be left off:

python3 -m pip install ex_pkg...

For example, the following command installs the wheel package, which is handy for building and working with Python wheel files. After creating a virtual environment, you may want to have it installed:

python3 -m pip install wheel

Since we are working with static sites for our example, we install pelican, a Python-powered static site generator, too:

python3 -m pip install pelican

A virtual environment can be deactivated from any file system location by entering deactivate, or by exiting the shell from which you activated the virtual environment.

Installed Packages

pip provides a great way to both record installed Python packages and to quickly re-install them. When combined with a virtual environment, this enables us to easily recreate virtual environments with the exact software versions installed when the virtual environment installed packages were first recorded.

We can save the installed Python packages to a specially formatted requirements.txt file using pip freeze:

python3 -m pip freeze > requirements.txt

python3 -m pip freeze | grep -v 'pkg_resources' > requirements.txt (Debian)

For example, if we wanted to do this for our ${HOME}/venvs/static_sites/ virtual environment, we would ensure that the environment was first activated and then run the following:

(static_sites) $ python3 -m pip freeze |
    grep -v 'pkg_resources' > requirements.txt

If we examined the requirements.txt file, it might look something like this:

blinker==1.5
docutils==0.19
feedgenerator==2.0.0
Jinja2==3.1.2
markdown-it-py==2.1.0
MarkupSafe==2.1.2
mdurl==0.1.2
pelican==4.8.0
Pygments==2.14.0
python-dateutil==2.8.2
pytz==2022.7.1
rich==13.2.0
six==1.16.0
Unidecode==1.3.6

If we or a collaborator need to recreate this project's virtual environment, we can do so with this requirements.txt file like so:

python3 -m pip install -r 'requirements.txt'

For example, if a collaborator downloaded our requirements.txt file to ${HOME}/Downloads/ and had a new ${HOME}/venvs/test_venv/ virtual environment they wanted to recreate our static_sites virtual environment in, they could run the following command after activating the test_venv virtual environment:

(test_venv) $ python3 -m pip install -r "${HOME}/Downloads/requirements.txt"

Data/Content Recommendations

A virtual environment may need to be something that is labile and/or ephemeral, so it may not be best to store your project data inside of it. However, the virtual environment software you install needs a way to access this data.

To achieve this, it may be best to store your project data outside of its virtual environment and to symbolically link to it from within the virtual environment. For example, we can store content for a static site at ${HOME}/sites/pelican_proj/ and then create a symbolic link to it inside of our ${HOME}/venvs/static_sites/ virtual environment:

ln -s ${HOME}/sites/pelican_proj/ ${HOME}/venvs/static_sites/pelican_proj

Now, if the static_sites virtual environment ever needs to be recreated, you just need to recreate the symbolic link for it to regain access to its data.

A Solid Foundation

Learning how to set up a development environment is often one of the most difficult and underappreciated parts of getting started with a new technology. There may not be a universally accepted way of developing with Python, but there are many commonly accepted best practices that can help you start out on solid ground.

Enjoyed this post?

Subscribe to the feed for the latest updates.