The test runner

The test runner collects and runs a number of test suites and writes all the results to report files.

Run the test runner

Running the test runner is as simple as:

from lily_unit_test import TestRunner

TestRunner.run("path/to/test_suites")

The test runner can be configured with a dictionary containing options. See the test runner API section for all available options and examples.

Collecting and running test suites

Test suites are recursively collected from the Python files in the given folder. Given the following project structure:

project_files
  |- src
  |   |- folder_01
  |   |   |- module_01.py
  |   |   |- module_02.py
  |   |
  |   |- folder_02
  |       |- module_03.py
  |       |- module_04.py
  |
  |- test
      |- test_runner.py

The test_runner.py contains the following code:

from lily_unit_test import TestRunner

TestRunner.run("../src")

The test runner is located in the test folder. The test runner will run all tests in the folder: ../src. This is relative to the test folder. Be sure to run the test runner from the test folder. You can also use an absolute path to the folder containing the test suites.

The test runner will scan all Python modules in the folder src recursively. This means all 4 python modules are checked for test suites.

The test runner imports each module and checks if the module contains a class that is based on the test suite base class (e.g.: class MyTestSuite(lily_unit_test.TestSuite)).

All test suites are executed in alphabetical order. If a specific order is required, use numbers in the file and folder names to sort them. The test runner will run all the test suites and will write report files to a folder. The output folder will look something like this:

project_files
 |- src
 |- tests
 |- lily_unit_test_reports                  // generic report folder
     |- 20231220_143717                     // date and time of the test run
         |- 1_TestRunner.txt                // test runner log
         |- 2_TestSuiteFromModule01.txt     // test suite log
         |- 3_TestSuiteFromModule02.txt     // test suite log
         |- 4_TestSuiteFromModule03.txt     // test suite log
         |- 5_TestSuiteFromModule04.txt     // test suite log

The first log file is from the test runner. This contains an overview of all test suites that are executed and their results. For each test suite a specific log file is created, containing all the messages from the test suite logger.

Test Runner API

class lily_unit_test.TestRunner

Static class that runs test suites in a specified folder.

classmethod run(test_suites_path, options=None)

Run the test suites that are found in the given path recursively.

Parameters:

• test_suites_path – path to the test suites

• options – a dictionary with options, if no dictionary is given, defaults are used

Returns:

True, if all test suites are passed

Options: The options dictionary can have the following values:

Key name	Default value	Description
report_folder	“lily_unit_test_reports”	The path where the reports are written. The path is by default at the same level as the test_suites_path. When setting a path, use an absolute path.
create_html_report	False	Create a single file HTML report.
open_in_browser	False	Open the HTML report in the default browser when all tests are finished.
no_log_files	False	Skip writing text log files. In case another form of logging is used, writing text log files can be skipped.
include_test_suites	[]	Only run the test suites in this list. Other test suites are skipped.
exclude_test_suites	[]	Skip the test suites in this list.
run_first	None	Run this test suite first.
run_last	None	Run this test suite last.

Not all keys have to present, you can omit keys. For the missing keys, defaults are used. For test suite names, use their class names.

Example: using HTML reporting and skip the text log files:

from lily_unit_test import TestRunner

options = {
    # Creates a single HTML file with all the results
    "create_html_report": True,

    # Open the HTML report in the default browser when finished
    "open_in_browser": True,

    # Do not write log files, because we use the HTML report
    "no_log_files": True
}
TestRunner.run(".", options)

Example: skipping test suites

from lily_unit_test import TestRunner, TestSuite

class MyTestSuite(TestSuite):
    # some test stuff

# options for the test runner:
options = {
    "exclude_test_suites": ["MyTestSuite"]
}
TestRunner.run(".", options)

Example: running only one test suite

from lily_unit_test import TestRunner, TestSuite

class MyTestSuite(TestSuite):
    # some test stuff

# options for the test runner:
options = {
    "include_test_suites": ["MyTestSuite"]
}
TestRunner.run(".", options)

Example: run specific test suites first and last

from lily_unit_test import TestRunner, TestSuite

class TestEnvironmentSetup(TestSuite):
    # Set up our test environment using test methods

class TestEnvironmentCleanup(TestSuite):
    # Clean up our test environment using test methods

options = {
    "run_first": "TestEnvironmentSetup",
    "run_last": "TestEnvironmentCleanup"
}
TestRunner.run(".", options)

Because the options are in a dictionary, they can be easily read from a JSON file.

import json
from lily_unit_test import TestRunner

TestRunner.run(".", json.load(open("/path/to/json_file", "r")))

This makes it easy to automate tests using different configurations.