Streamline AI/ML Project Documentation on Jupyter notebook & ReadTheDocs

Automate Jupyter Notebook Conversion with Sphinx on ReadTheDocs and never worry about documentation again

Photo by Catherine Heath on Unsplash

In the fast-paced world of AI and ML development, clear and comprehensive documentation is a cornerstone of success. It’s the key to ensuring that your project is understandable, maintainable, and able to be seamlessly handed off to collaborators or stakeholders.

Jupyter notebooks have become indispensable tools for data scientists and engineers, serving as dynamic environments for experimentation and code development.

However, when it comes to creating project documentation for AI and ML endeavours, these notebooks often fall short.

In this blog post, we’ll explore a powerful solution: automatically converting Jupyter notebooks into polished project documentation hosted on ReadTheDocs using Sphinx. This streamlined process will not only save you time but also enhance the accessibility and professionalism of your project’s documentation.

We’ll begin by creating Bitbucket and ReadTheDocs accounts. Next, we’ll initialize a Sphinx project and add the necessary extensions for Jupyter notebooks. Finally, we’ll host everything on ReadTheDocs.

Creating BitBucket Repository

Create a Bitbucket account if you don’t have one and create a new public repository. I’ve opted for a free ReadTheDocs account (community edition), which exclusively supports public repositories. To host your private repo’s documentation, you’ll need to create a company ReadTheDocs account.

An empty initialized repo will look like this:

Clone this repo in your local machine to make changes in the code.

Activate your pip or conda environment, I recommend conda. Make sure it has shpinx and python installed.

conda create --name sphinx-blog python=3.9 sphinx pip pandas ipykernel ipywidgets

conda activate sphinx-blog

Initialize a default sphinx project

Set up a source directory and creates a default conf.py with the script called sphinx-quickstart:

sphinx-quickstart

It will ask you for some inputs like:

1. Separate source and build directories (y/n) [n]
2. Project name
3. Author name(s)
4. Project release
5. Project language

You will see the “Finished” message once it is done executing.

Navigate to the project directory on your local machine and open it in your code editor. Inside, you’ll find several new files and two crucial directories: : build and source.

In this example, we’ll stick with the defaults, but you can customize them as needed.

Now run the build with the sphinx-build program:

sphinx-build -b html source build

You will see `build` directory has numerous new files including some html pages.

Open the index.html page in your browser

Connecting BitBucket repo with ReadTheDocs

Create a ReadTheDocs account, log in, and then click on “Connect your Accounts” from the dashboard’s side panel.

Since I am using a Bitbucket repository, we will be connecting our ReadTheDocs account to BitBucket. However, you can also connect it to GitHub or GitLabs if you prefer.

Login in to BitBucket account:

Click on “Grant Access”

From the dashboard, import the relevant repository. If your public repositories do not automatically appear on the dashboard, click the refresh button and reload the page.

Enter the project details that will be displayed on the public documentation page, and then click “Next.”

Copy this code and paste it into a new .readthedocs.yml file, which should be created in the root directory of your project.

And on the ReadTheDocs page, click on “Next” to finish the import.

In the code editor, edit the .readthedocs.yml file and change line number 20 to reflect the correct source configuration. In our case, it should be source/conf.py.

Now, commit all the files in your Git repository and push the changes to the remote main branch (or master).

Back in your ReadTheDocs dashboard, under builds tab, you will see the latest build.

You can click on latest build to monitor the progress step by step. Once it is completed, it should be marked as “Passed.” If it shows any errors (i.e. “Failed”), use the messages to debug the issue.

Back in your ReadTheDocs dashboard, go to the “Overview” tab, click on “latest,” and it will take you to your project’s public documentation.

Now we will add jupyter notebook support.

Jupyter notebook documentaion

Public repo link: https://bitbucket.org/jbg95866/jupyternb/src/main/

Replace source/index.rst with following snipper:

Your AI/ML Project Documentation
==================================

.. toctree::
   :maxdepth: 2

   docs/index

Create a new directory docs inside `source` directory, and add two files index.rst and basic.ipynb .

The index.rst inside source/docs should resemble the screenshot above, and the code is provided below.

Exploration data analysis
=========================

.. toctree::
   :maxdepth: 2

   basic

Additionally, the basic.ipynb can be a simple notebook with a few cells containing basic outputs.

Update the code in your source/conf.py file with the following code to enable Jupyter notebooks support:

project = "JNBDoc"
copyright = "2023, Mahesh Rajput"
author = "Mahesh Rajput"
release = "0.0.1"

# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

extensions = [
    # https://myst-nb.readthedocs.io
    "myst_nb",
]
source_suffix = [".rst", ".md"]

templates_path = ["_templates"]
exclude_patterns = []

language = "English"

# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output

html_theme = "sphinx_rtd_theme"
html_static_path = ["_static"]

# myst-nb plotly
html_js_files = [
    "https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.4/require.min.js"
]


# -- Options for warnings -------------------------------------------------

# Need for MyST-NB extension
suppress_warnings = ["mystnb.unknown_mime_type"]


# -- Options for MyST-NB extension -------------------------------------------------

# Turn off notebooks execution
nb_execution_mode = "off"


myst_heading_anchors = 4

Now create a build using following command:

sphinx-build -b html source build

It will give you some errors.

Because we haven’t installed my-bd extension. Use pip to install the extension:

pip install myst-nb

Now run:

sphinx-build -b html source build

Go the index.html file in your build directory.

You can click on “EDA” link and you will see your jupyter notebook outputs.

You can also navigate directly to a particular section of your notebook (or documentation).

To make it more interesting install “sphinx-rtd-theme” using

pip install sphinx-rtd-theme

Run build command again and check files locally:

sphinx-build -b html source build

Reflecting the same changes on readthedocs

To enable same extension and theme reflects on readthedocs:

1. Add a docs_env.yml in the root directory of project

name: rtd38
channels:
  - defaults
  - conda-forge
dependencies:
  - python=3.9
  - pip=23.2.1
  - sphinx=5.3.0
  - sphinx_rtd_theme=1.3.0
  - pip:
    - myst-nb==0.17.2
    - myst-parser==0.18.1

2. Add environment details in your .readthedocs.yml file:

conda:
  environment: docs_env.yml

And change the build object to this:

build:
  os: ubuntu-22.04
  tools:
    python: "mambaforge-4.10"

Final file is in public repo:

jupyternb Bitbucket Repo

Now commit and push the changes to remote bitbucket branch (you must also add build directory to .gitignore )

Check the build status:

Once build passes, go back to your public project documentation link

Thats it!

Harnessing the power of Sphinx to seamlessly integrate Jupyter notebooks into your project documentation on ReadTheDocs can be a game-changer for AI/ML projects.

It streamlines the process, enhances accessibility, and ensures that your project’s valuable insights are effectively communicated. By automating this workflow, you not only save time but also empower your team and stakeholders with up-to-date, easily digestible and maintanable project documentation.

You can connect with me on LinkedIn: https://www.linkedin.com/in/maheshrajput/

Welcome - Sphinx documentation

Full featured documentation deployment platform

jupyternb Bitbucket Repo