Hello, Blog!

If you have just stumbled upon my OSD600 series of blog posts, it has been created to document and share my learnings as I progress through my Open Source Development college course.

Aside from continuing to work on contributing to Open Source projects, this week's focus was on looking into "managing project complexity through the use of Automated Testing" & starting to write test cases for our CLI tools.

Testing is a fundamental aspect of software development. Tests allow us to refine the code's logic, ensuring each component performs as expected even when handling edge cases. Automated tests make it easier to introduce changes to code without breaking its core logic, which is especially important in the Open Source project (as they rely on collaboration).

Helpful links:

• A Beginner's Guide to Unit Testing with Pytest


• Mastering Unit Tests in Python with pytest: A Comprehensive Guide


• pytest documentation


• pytests-mock documentation


• My Project Repository

pip install .

I welcome feedback/contributions, so don’t hesitate to take a look and get involved 😊

Starting to write tests in Python

Beforehand, I had only written tests using Jest in one of my Node.js projects, so I had a general understanding of how to approach testing. However, I had no experience with testing in Python.

So, to start, I went "shopping" for the right testing library. After reading a couple of articles and community discussions, I decided to begin writing tests using Pytest in combination with the pytest-mock plugin, which I used to mock the API interactions with the LLM.

Why Pytest?

Pytest is one of the most popular testing frameworks that combines simple syntax (e.g. less boilerplate code compared to Python's built-in unittest library) with powerful features and a vast ecosystem of plugins for nearly every testing scenario.

Prep & Setup

Before diving straight into the documentation, I like to go through a few simple beginner guides to get a quick overview of the tools and their functionality. I did the same with Pytest, reading two beginner guides before jumping into setup and coding (A Beginner's Guide to Unit Testing with Pytest, Mastering Unit Tests in Python with pytest: A Comprehensive Guide).

The setup was close to none, I simply installed the library and the mocking plugin using pip, created a new tests/ folder at the root of my project, and was good to go:

pip install pytest pytest-mock

mkdir tests/

Writing the first test

I began testing with the simplest function in my project: the version_callback() function, which prints the version of the CLI tool when the --version/-v flag is provided.

I covered three test cases:

• Successful version output – Verifying that the version prints correctly to the terminal when the flag is used.


• No flag behaviour – Ensuring that not providing the flag does not cause any unexpected behaviour.


• Handling unset version – Confirming that the function prints an appropriate message if the version is not set.

Writing the tests was straightforward, I mostly followed examples from guides and documentation, with occasional clarifications from an AI chatbot.

I used pytest assertions for stdout comparisons and pytest.raises for exception checks and also got to try out the patching to replace the imported version value with None.

import pytest

from typer import Exit

from app.core.callbacks import version_callback

from app import __version__





def test_version_callback_without_version_defined(mocker, capfd):

    """

    Test version callback function when the version is not defined

    """

    mocker.patch("app.core.callbacks.__version__", None)

    with pytest.raises(Exit):

        version_callback(provided=True)



    # Capture the output

    captured = capfd.readouterr()



    # Check that it prints a fallback message instead of the version

    assert "version not defined" in captured.out.strip()

See the full code on GitHub

Testing the core functionality/Mocking LLM interactions

The core functionality of the tool revolves around interacting with the LLM, which is handled in the generate_comments() function. To test this, I needed to mock the LLM interaction.

To simulate the OpenAI client interaction, I used the pytest-mock plugin with its mocker fixture. This allowed me to mock the behaviour of the OpenAI client and control the responses returned by its methods during testing.

In pytest, a fixture is a function that prepares and provides the necessary setup for tests. Fixtures are defined with the @pytest.fixture decorator and can be automatically passed into test functions as arguments.

@pytest.fixture

def mock_openai_client(mocker):

    """

    Fixture to mock the OpenAI client and its responses

    """

    # Create a mock object with MagicMock()

    mock_client = mocker.MagicMock()

    # Replace the client with the mock object whenever 

    # OpenaAI() is called in the code

    mocker.patch("app.core.api.OpenAI", return_value=mock_client)

    return mock_client



####################################################################



def test_generate_comments_with_api_key(test_parameters, mocker, mock_openai_client):

    """

    Test that the function generates expected output when a valid file path and API key are provided

    """

    file_path, content = test_parameters



    commented_code = """

                    def generate_fibonacci(limit):

                        # Initialize the Fibonacci sequence with the first two numbers

                        fibonacci_sequence = [0, 1]

                        # Generate Fibonacci numbers until the next number exceeds the given limit

                        while True:

                            # Calculate the next number in the sequence

                            next_number = fibonacci_sequence[-1] + fibonacci_sequence[-2]

                            # Stop generating if the next number exceeds the limit

                            if next_number > limit:

                                break

                            # Append the next number to the sequence

                            fibonacci_sequence.append(next_number)

                        return fibonacci_sequence



                    """



    # Replicate the OpenAI client response structure using mock objects

    mock_response = mocker.MagicMock()

    mock_response.choices = [

        mocker.MagicMock(message=mocker.MagicMock(content=commented_code))

    ]

    # Set the result of the mock chat completion to the mock response,

    # so when the value is accessed in the code it gives the expected result

    mock_openai_client.chat.completions.create.return_value = mock_response



    result = generate_comments(

        file_path, content, None, "test_api_key", None, None, False

    )



    assert result == commented_code

See the full code on GitHub

Afterthoughts

From the start, I tried making my code clean and modular by breaking it down into smaller functions. This approach certainly made my project more testable. But, along the way, I realized that I hadn't fully accounted for all possible edge cases ("Wait, does the logic break when this value is None?"). From now on, I will try to look at my code more critically by asking more "what if?" questions.

Writing the first few tests helped me gain a solid understanding of pytest. I learned how to do simple assertions, test exceptions, capture data written to the terminal, use fixtures for setup, and how to replace values and mock different objects using mocker.

Although testing the generate_comments() and version_callback() functions has given me a good foundation, I still need to improve test coverage, explore testing argument parsing and figure out a way to test the main entry point function.