Page Body

Implement a Personal Documentation System That Maximizes Flexibility and Freedom

Table of Contents

One great aspect of some of the most successful and long-lived Free/Libre Open Source (FLOSS) projects is the high quality of their documentation. Well-structured, approachable, and attractive documentation can help you decide whether you have found the right solution, provide an edification opportunity, and serve as an ongoing reference for future issues you may encounter.

We live in an information-dense world, and as we continually learn, it is helpful to keep track of and organize the knowledge we accumulate. Creating a personal documentation system is an excellent way of accomplishing this.

As a GNU/Linux user, you have likely already spent significant time becoming versed in developer-oriented technologies, including:

The same tools provide the base for a powerful and convenient approach to documentation, referred to as Documentation as Code (Docs as Code).

Note: If you are not familiar with the GNU/Linux command line interface, review the Conventions page before proceeding.

Static Sites via Docs as Code

Docs as Code refers to a philosophy of writing documentation with the same tools and methodologies that are used to write code. The basic idea is that content is written in a user-friendly and accessible markup language (e.g., Markdown, reStructuredText), tracked via a version control system (Git), and then automatically tested/published to a serving resource.

Often, a static site generator is used to convert the plain text markup into a final product that can be locally viewed or pushed to a remote web server. There are many FLOSS static site generators to choose from that are based on different technologies, but they all share a few basic characteristics:

  • Content is created in a plain text markup language.
  • Templates are used to provide the HTML structure for the final site.
  • A CLI tool is used to inject the content into the template files and generate the final site.
  • How and where you deploy your final site is up to you.
  • Extensions/plugins are available to add functionality.

This paradigm differs from many of the traditional content management systems (CMSs) that are database-driven web applications. By utilizing a static site generator and a Docs as Code approach, you can avoid many traditional CMS drawbacks:

  • Improved security posture.

    Compared to a database-driven web application, a static website has a much simpler setup with fewer dependencies. This means less surface area for vulnerabilities, less time and effort to get to a secure state, and an easier ongoing experience in maintaining secure content delivery.

  • Improved performance.

    While web applications can be configured to serve static content, they often dynamically generate requested HTML pages in real-time. With a static site, your server is simply serving your content, which often leads to better performance for content consumers.

  • Reduced administrative burden.

    With a web application, there is likely a complicated stack of software that needs to be continually monitored, updated, and troubleshot. On top of this, you will need to monitor, update, and troubleshoot the CMS itself (along with its themes and plugins).

    If you are using a static site, you are basically maintaining a web server, and even that can be handed off to a third-party provider, if you choose to. There is the local stack used to generate your static site to deal with, but that is likely much easier to maintain (and already being taken care of through the administration of your GNU/Linux system).

  • Retention of dynamic elements.

    Just because a site is static does not mean that it cannot deliver a dynamic experience. Compared to a database-driven web application, dynamic functionality for a static site is confined to the client side (i.e., the end user's web browser). Through technologies like CSS, JavaScript, and WebAssembly, rich, interactive static sites can be created, without the security, performance, and administrative baggage of traditional CMSs.

  • Data accessibility.

    With a static site generator, your content is just a collection of plain text files and assets on your local file system. Since content it not being stored in a database management system (DBMS), it always remains easy to access with whatever tool you desire. Being able to open your favorite editor and simply view your content is a nice alternative to using a What You See Is What You Get (WYSIWYG) editor or running a Structured Query Language (SQL) query.

Static sites via Docs as Code are not always the best solution. If the CMS needs to be used by numerous people that are not comfortable with a developer-oriented toolchain, you are dealing with large amounts of media assets, or you need a well-developed WYSIWYG editor and graphical dashboard, you may be better off with a traditional CMS.

However, for a personal documentation system, adopting Docs as Code and a static site generator is an excellent choice.

Choosing a Static Site Generator

Two very popular static site generators for maintaining a document store are MkDocs and Sphinx. Both are Python-based and use Jinja as their templating engine. Either one of these projects is a fine solution for a personal documentation system, but there are some differences.

MkDocs is based on Markdown and is simpler to get started with. Sphinx is based on reStructuredText (but has excellent Markdown support via MyST), has greater functionality/extensibility, but has a slightly higher learning curve.

Another important difference is related to maintenance. MkDocs is maintained by a relatively small number of people. Sphinx is a slightly older project with a larger maintenance team, perhaps in part because it has been adopted as the documentation generator of choice for numerous FLOSS projects, including:

If you decide that MkDocs is a better fit for your documentation needs, these resources may help you get started:

There are also several good personal and professional examples of MkDocs usage in the wild:

Examining how these sites are set up may give you ideas on how to best customize your MkDocs implementation.

If you are curious about Sphinx, but are not sure if it is worth the slightly higher initial investment time, the following presentation by Paul Everitt does a great job of explaining the power and flexibility of the software:

Static Sites with Sphinx and Markdown

The following sections will walk through a Sphinx setup, but (outside of project-specific commands) most of it will apply to any Python-based static site generator.

The Virtual Environment

Sphinx is a Python-based static site generator, so after making sure that pip is set up, our first step (as with any Python-based project) is to create a virtual environment. We create our virtual environment at ${HOME}/venvs/sphinx/:

$ python3 -m venv --upgrade-deps "${HOME}/venvs/sphinx/" &&
    cd "${HOME}/venvs/sphinx/" &&
    source 'bin/activate' &&
    python3 -m pip install wheel

The above commands do several things:

  1. Create a virtual environment at ${HOME}/venvs/sphinx/ and update its core dependencies (i.e., pip and setuptools).
  2. Change to the virtual environment directory.
  3. Activate the virtual environment.
  4. Install wheel in the virtual environment.

Installing Packages

Next, we need to install the software packages in our virtual environment needed for our Sphinx site. In addition to the Sphinx package, there are several other packages to install:

  • myst-parser - MyST is a rich and extensible flavor of Markdown meant for technical documentation and publishing. This package enables us to elegantly use Markdown with our Sphinx site. In addition, it maps new Markdown syntax to Sphinx-specific features, thus enabling us to continue using Markdown, while simultaneously gaining the additional functionality of Sphinx.
  • sphinx-book-theme - Sphinx comes with a great default theme, Alabaster, but there are many other nice Sphinx themes to choose from. Sphinx Book Theme is one of them.
  • sphinx-autobuild - This package rebuilds Sphinx documentation on changes with live-reload in the browser. Essentially, this lets you preview the site in the browser as you work on your documentation changes.
  • sphinx-copybutton - Often, you will see a handy option to copy the contents of a code block as you peruse documentation. sphinx-copybutton adds this functionality to Sphinx sites.

To install these packages in our virtual environment, we run:

(sphinx) $ python3 -m pip install \
    myst-parser \
    Sphinx \
    sphinx-autobuild \
    sphinx-book-theme \
    sphinx-copybutton

Data Isolation

Virtual environments are often ephemeral, so we do not want to store our Sphinx content inside of it. So, our next step is to create a docs/ directory in a more persistent location, and then create a symbolic link to it in the virtual environment directory:

(sphinx) $ mkdir -p "${HOME}/venv_projs/docs/" &&
    ln -s "${HOME}/venv_projs/docs/" "${HOME}/venvs/sphinx/docs"

Project Creation

A Sphinx project has a specific structure, and Sphinx comes with an interactive script that automatically creates this structure for you, sphinx-quickstart. We change to the directory where we want to create our Sphinx project and run the script:

cd "${HOME}/venvs/sphinx/docs/" && sphinx-quickstart

After running the above commands, we are asked several questions regarding how to configure the project. We answer like so:

  • > Separate source and build directories (y/n) [n]: y
  • > Project name: Docs
  • > Author name(s): Monty
  • > Project release []:
  • > Project language [en]:

Afterwards, the requisite directories/files will be created and you should see a message that starts with a line like:

Finished: An initial directory structure has been created.

Let us quickly look at the created directory tree:

docs
├── build
├── make.bat
├── Makefile
└── source
   ├── conf.py
   ├── index.rst
   ├── _static
   └── _templates
build/
The directory that holds the rendered content.
make.bat, Makefile
Convenience scripts to simplify some common Sphinx operations (e.g., rendering the content).
source/conf.py
The Python script holding the configuration of the project. This file contains the project name and release specified during sphinx-quickstart, as well as additional configuration keys.
source/index.rst
The root document of the project. source/index.rst serves as the welcome page and contains the root of the table of contents tree (toctree), i.e., a way to connect multiple files into a single hierarchy of documents.
source/_static/
The directory for custom stylesheets and other static files.
source/_templates/
The directory for custom HTML templates.

Project Configuration

Before we build and serve the site, let us take a look at the default source/conf.py file:

# Configuration file for the Sphinx documentation builder.
#
# For the full list of built-in configuration values, see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html

# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information

project = 'Docs'
copyright = '2023, Monty'
author = 'Monty'

# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

extensions = []

templates_path = ['_templates']
exclude_patterns = []



# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output

html_theme = 'alabaster'
html_static_path = ['_static']

Via source/conf.py, we can use Python constructs like variables, strings, and lists to make configuration changes to our Sphinx site. The Project information section looks OK, but we need to make some changes to the General configuration and Options for HTML output sections.

Afterwards, source/conf.py should look like this:

# Configuration file for the Sphinx documentation builder.
#
# For the full list of built-in configuration values, see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html

# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information

project = 'Docs'
copyright = '2023, Monty'
author = 'Monty'

# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

extensions = ['myst_parser', 'sphinx_copybutton']

templates_path = ['_templates']
exclude_patterns = []

myst_enable_extensions = [
    'deflist',
]
myst_heading_anchors = 6

# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output

html_theme = 'sphinx_book_theme'
html_static_path = ['_static']
html_title = 'Docs'

General Configuration

  • In the extensions list, we add the module names of the Sphinx extensions we want to use.
  • myst_enable_extensions is one of many MyST global configuration variables. Specifically, we use it to enable definition lists.
  • myst_heading_anchors specifies the depth to which Markdown headings should be converted into HTML anchor links.

Options For HTML Output

  • We change html_theme from the default theme value to our new theme module name, sphinx_book_theme.
  • The title of a Sphinx project defaults to <project> v<revision> documentation. We use html_title to change it to the simpler Docs.

Building and Serving the Site

Before we start adding content, let us see what we have so far looks like by building and serving the site with sphinx-autobuild from the ${HOME}/venvs/sphinx/docs/ directory:

(sphinx) $ make clean &&
    sphinx-autobuild \
        -b html -j auto \
        "${HOME}/venvs/sphinx/docs/source/" \
        "${HOME}/venvs/sphinx/docs/build/"

make clean wipes the build/ directory before sphinx-autobuild builds the site. This ensures that all of your changes are reflected in the new build.

sphinx-autobuild's -b and -j options are passed as is to the underlying sphinx-build command, which sphinx-autobuild utilizes (run sphinx-build --help for more information):

  • -b html is used to specify the builder to use (here, we are building html output files).
  • -j auto builds the site with N processes, where N is equal to your system's central processing unit (CPU) count (this should yield faster build times).

Then, we provide sphinx-autobuild with the paths of the source/ and build/ directories, respectively. After we run the command, we can view our site at http://127.0.0.1:8000/.

The Sphinx Book Theme is a very nice responsive theme that has helpful features common to many other popular Sphinx themes, including a light/dark theme toggle, built-in search, and dedicated navigation areas (both for the site as a whole and for the currently viewed document).

The TOC Tree

Sphinx offers many of its features through directives, generic blocks of explicit markup that can have:

  • Arguments - Given directly after the colon following the directive's name. Each directive decides whether it can have arguments, and how many.
  • Options - Given after arguments in the form of a field list.

  • Content - Follows the options or arguments after a blank line. Each directive decides whether to allow content, and what to do with it.

    Make sure that the first line of content is indented to the same level as the options.

Sphinx is built around reStructuredText (reST), which cannot interconnect several documents or split documents into multiple output files. To address this, Sphinx uses a custom directive, toctree, to add relations between the single files the documentation is made of, as well as tables of contents.

We can see what this looks like by examining the default source/index.rst file that was created for us:

.. Docs documentation master file, created by
   sphinx-quickstart on Sat Mar 18 17:40:39 2023.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Welcome to Docs's documentation!
================================

.. toctree::
   :maxdepth: 2
   :caption: Contents:



Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

source/index.rst starts off with an explicit markup block (i.e., a line that starts with ..) that is not followed by a valid markup construct. This is how a comment is created in reST:

.. Docs documentation master file, created by
   sphinx-quickstart on Sat Mar 18 17:40:39 2023.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

The above example is also a multiline comment, which is created by indenting text after a comment start.

After the comment is the index page's first section header. Section headers are created by underlining (and optionally overlining) the section title with a punctuation character, at least as long as the text:

Welcome to Docs's documentation!
================================

Normally, there are no heading levels assigned to certain characters, as the structure is determined from the succession of headings. However, this convention is used in Python's Style Guide for documenting, which you can adopt:

  • # with overline, for parts
  • * with overline, for chapters
  • = for sections
  • - for subsections
  • ^ for subsubsections
  • " for paragraphs

You can use your own marker characters and use a deeper nesting level, but remember that most target formats (e.g., HTML, LaTeX) have a limited supported nesting depth.

Next, we come to the toctree directive:

.. toctree::
   :maxdepth: 2
   :caption: Contents:

This toctree directive block does not have any arguments or any content, which is why the navigation area for the site is empty. However, we do see two options, maxdepth and caption.

  • maxdepth is used to indicate the depth of the tree. By default, all levels are included.
  • caption is used to provide a toctree caption.

The final section of source/index.rst contains a list of internal subsection links. Internal linking to specific subsections is done via a special reST role provided by Sphinx, :ref::

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

These internal links use special names and are for the general index, Python module index, and the search page.

Building the Documentation System

Often, Sphinx is used to document code-based projects. Since we aim to create a personal documentation system that includes code (but is not specifically documenting code), let us simplify source/index.rst to something more appropriate.

First, we create some new directories in ${HOME}/venvs/sphinx/docs/source. Assume that we intend on creating documents geared towards science and technology, and we want those files to be collected into separate directories. Also, we want them to be available in our documentation site under different navigational sections:

(sphinx) $ mkdir -p \
    "${HOME}/venvs/sphinx/docs/source/science/biology/" \
    "${HOME}/venvs/sphinx/docs/source/technology/computer-science/"

Next, we create example documents in each directory. Since Sphinx and MyST allow us to mix reST and Markdown content, we create each document as a Markdown document:

(sphinx) $ echo -e '# Science Notes\n\nExample _science_ notes.' > \
        "${HOME}/venvs/sphinx/docs/source/science/sci-notes.md" &&
    echo -e '# Ecology Notes\n\nExample _ecology_ notes.' > \
        "${HOME}/venvs/sphinx/docs/source/science/biology/ecology.md" &&
    echo -e '# Technology Notes\n\nExample _technology_ notes.' > \
        "${HOME}/venvs/sphinx/docs/source/technology/tech-notes.md" &&
    echo -e '# Algorithm Notes\n\nExample _algorithm_ notes.' > \
        "${HOME}/venvs/sphinx/docs/source/technology/computer-science/algo-notes.md"

For each directory we create, we also create a separate file that serves as an index page for that directory. Each of these pages will contain its own toctree.

These pages will be included in source/index.rst's toctree, which will enable us to nicely navigate our document structure via our theme. Documents serve as the toctree directive content and are given as document names, i.e., leave off the filename extensions and use forward slashes (/) as directory separators.

We can create these files as reST or Markdown files, and how you choose to set up your documentation system is up to you. For example, this is how a toctree could be set up for our science directory using MyST:

```{toctree}
:maxdepth: 1

biology/biology
sci-notes
```

Depending on your preference and the construct in question, the syntax for reST or Markdown will seem preferable. For now, we will continue creating toctrees using reST.

We create the following four files:

  1. ${HOME}/venvs/sphinx/docs/source/science/science.rst

    Science
=======

.. toctree::
   :maxdepth: 1

   biology/biology
   sci-notes

  2. ${HOME}/venvs/sphinx/docs/source/science/biology/biology.rst

    Biology
=======

.. toctree::
   :maxdepth: 1

   ecology

  3. ${HOME}/venvs/sphinx/docs/source/technology/technology.rst

    Technology
==========

.. toctree::
   :maxdepth: 1

   computer-science/computer-science
   tech-notes

  4. ${HOME}/venvs/sphinx/docs/source/technology/computer-science/computer-science.rst

    Computer Science
================

.. toctree::
   :maxdepth: 1

   algo-notes

After these changes, we end up with a source/ directory structure like so:

source
    ├── conf.py
    ├── index.rst
    ├── science
    │   ├── biology
    │   │   ├── biology.rst
    │   │   └── ecology.md
    │   ├── science.rst
    │   └── sci-notes.md
    ├── _static
    ├── technology
    │   ├── computer-science
    │   │   ├── algo-notes.md
    │   │   └── computer-science.rst
    │   ├── technology.rst
    │   └── tech-notes.md
    └── _templates

Finally, we need to update our source/index.rst file to include science.rst and tecnology.rst, and to do some customization (e.g., the hidden option makes sure that the toctree is not shown on the source/index.rst page content area itself, since our theme will take care of displaying it in the navigation area):

.. Docs documentation master file, created by
   sphinx-quickstart on Sat Mar 18 17:40:39 2023.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Personal Documentation System Demo
==================================

.. toctree::
   :caption: Table of Contents
   :hidden:
   :maxdepth: 2

   science/science
   technology/technology

Utilize the **power** of Sphinx and retain the *simplicity* of Markdown by using `MyST <https://myst-parser.readthedocs.io/en/latest/index.html>`_.

If you were already previewing the site, press Ctrl+c to terminate the process and run it again to view the changes:

(sphinx) $ make clean &&
    sphinx-autobuild \
        -b html -j auto \
        "${HOME}/venvs/sphinx/docs/source/" \
        "${HOME}/venvs/sphinx/docs/build/"

Version Control

The basic structure of the documentation system is complete. You are the only person working on the documents, but before adding more content, you want to:

  • Implement version control (Git)
  • Keep everything on your local system
  • Keep things simple
  • Be able to work and track your changes offline

Also, you do not want to have the administrative burden of maintaining a Git forge, like Gitea or GitLab.

You can accomplish this by initializing a local directory on your system as a bare repository, which is a repository without a working directory. For example, assuming that you have already configured Git and are using main as the default branch name (git config --global init.defaultBranch main), we create ${HOME}/sites/docs.git/ and initialize it as a bare directory:

$ mkdir -p "${HOME}/sites/docs.git/" &&
    cd "${HOME}/sites/docs.git/" && git init --bare

Then, we can set up ${HOME}/venvs/sphinx/docs/source/ with Git:

$ cd "${HOME}/venvs/sphinx/docs/source/" &&
    git init && git add . && git commit -m 'Initial commit' &&
    git remote add origin "${HOME}/sites/docs.git/" &&
    git push --set-upstream origin main

Now, you can work on your documentation system, preview your work, and version control it in a self-contained, offline fashion.

Publishing Changes

When using a static site, publishing your content is a relatively easy endeavor. Essentially, you need to serve the contents of the build/ directory. Usually, this means syncing the contents of this directory after you create a new build of your site with a web server directory that you have configured to serve content.

GNU/Linux makes available wonderful tools for securely synchronizing changes like this over a network connection, ssh and rsync.

An example build/sync workflow could look like this:

  1. Add, commit, and push your changes:

    cd "${HOME}/venvs/sphinx/docs/source/" &&
    git add . && git commit -v && git push

  2. Activate the virtual environment:

    source "${HOME}/venvs/sphinx/bin/activate"

  3. Change to the docs/ directory:

    cd "${HOME}/venvs/sphinx/docs/"

  4. Run the build commands:

    (sphinx) $ make clean &&
    sphinx-build \
        -b html -j auto \
        "${HOME}/venvs/sphinx/docs/source/" \
        "${HOME}/venvs/sphinx/docs/build/"

  5. Sync the changes (note that rsync needs to be installed on both the local and remote system):

    $ rsync \
    -aHAXxz \
    --delete \
    -e "ssh -p ex_port" \
    --filter='-x security.selinux' \
    --info=progress2 \
    --numeric-ids \
    "${HOME}/venvs/sphinx/docs/build/" \
    ex_user@ex_host:ex_dest/"

Automation

You have walked through the process of creating, version tracking, building, and publishing content using a static site generator. However, an important part of the Docs as Code approach we have not yet incorporated is automation.

The benefits of static site generators over traditional CMSs are manifold, but continually entering these commands is onerous and error prone. Automation is preferable.

As a GNU/Linux user, you have likely already invested the time into learning the foundation of the Docs as Code stack. You are ready to create your own Docs as Code automation solution.

All that you have seen in this post is essentially a series of sequential commands, which means that it is scriptable. In addition, outside of the specific static site generator commands used in the examples, the basic flow is generic and common to other static site generators.

Imagine creating a series of scripts that can be independently used or chained together that cover the entire Docs as Code process. For example, there could be a script to:

  • Build and preview the site
  • Add, commit, and push site changes to the version control system (Git)
  • Build and publish the site

Whichever static site generator you decide to use, if you ever move to a different piece of software, you should be able to update the software-specific build/preview commands and continue using the scripts.

Your scripts can provide the basis for a versatile, stable, long-lived automation solution.

Docs or It Didn’t Happen

Organized, well-maintained documentation can be a boon and help us make sense of the deluge of information we are exposed to. By committing yourself to GNU/Linux and FLOSS, you are empowered with the tools you need to get started with the Docs as Code paradigm.

Enjoyed this post?

Subscribe to the feed for the latest updates.