The test suite

This page describes more details about the test suite class.

The test suite class is the main class for running tests. Each test case is defined as a method in the test suite. The method must start with test_. These test methods are executed when the test suite is executed.

Preceding the test methods, a setup method is executed. If the setup fails, execution is stopped. Following the test methods a teardown method is executed. The teardown method is always executed, regardless whether the test methods passed or failed.

Test suite creation

Creating a test suite is as simple as creating a subclass:

import lily_unit_test

class MyTestSuite(lily_unit_test.TestSuite):
    # My test suite

Test methods are added by adding methods with the prefix: test_:

import lily_unit_test

class MyTestSuite(lily_unit_test.TestSuite):

    def test_login(self):
        # test log in

    def test_upload_image(self):
        # test uploading image

In this case two test methods are defined. The test methods are executed in the order as they are created, from top to bottom.

Other methods can also be added to the test suite to provide specific functionality.

import lily_unit_test

class MyTestSuite(lily_unit_test.TestSuite):

    def connect_to_server()
        # connect to server

    def test_login(self):
        self.connect_to_server()
        # test log in

    def test_upload_image(self):
        self.connect_to_server()
        # test uploading image

In this test suite we added a helper method to connect to the server. We use this in each test method to connect to a server before doing the tests. The connect to server method, does not start with test_ and is ignored by the test suite when it is executed.

Running the test suite

The test suite can be executed using the run method. The run method returns True if the test suite passed and False if failed. In order to make the test suite run properly, the test suite must be initialized:

# Initialize test suite, the test suite require any parameters
ts = MyTestSuite()
# Run the test suite
ts.run()

# A nice one liner
MyTestSuite().run()

# Using the test result
if MyTestSuite().run():
    print("Yay, the test suite passed!")
else:
    print("Oops, the test suite failed...")

Using setup and teardown

The test suite has a default setup and teardown methods that can be overridden in the subclass. The default setup and teardown do nothing, they are just empty methods. If not overridden, it will not matter. The setup and teardown can be overridden in your test suite:

import lily_unit_test

class MyTestSuite(lily_unit_test.TestSuite):

    connection = None

    def setup(self):
        self.connection = connect_to_server(user, password)

    def test_upload_image(self):
        self.connection.upload_image(filename)

    def test_download_image(self):
        self.connection.download_image(uri, filename)

    def teardown(self):
        # In case the connection could not be created, the connection property could still be None
        if self.connection is not None and self.connection.is_connected():
            self.connection.close()

In this hypothetical example, prior to all tests a connection to a server is created in the setup method. In case this fails because of an exception, the execution stops and the test suite fails. In case the setup method passes, the test methods will be executed. Finally, the teardown is executed. The teardown closes the connection with the server. If in the hypothetical case, the connection was not established in the setup (failed for some reason), closing a not established connection can cause an exception. The test suite will fail if the teardown fails because of an exception.

Making test suites pass or fail

A test method or setup method is passed by the following conditions:

• There were no exceptions or asserts.

• There were no messages from the standard error handler (stderr).

• The return value is None (default return value of a method) or True.

A test method or setup method is failed by the following conditions:

• An exception or assert was raised

• There were messages from the standard error handler (stderr).

• The return value is False

The teardown method can only fail if an exception or assert was raised. The return value is not used.

The return value of a method in Python is by default None. If the test method is executed and the return value is None, the test method is marked as passed. If you wish to explicitly make a method fail, you can return False. The test suite will mark the test method as failed.

The test suite checks for messages from the standard error handler (stderr). There can be threads running in the background that generate exceptions. These exceptions cannot be caught by the test suite. But these exceptions will generate messages to the standard error handler. These messages are used for the test suite result.

Examples of passing or failing test suites

The following examples only show the specific test method from the test suite.

# Fails in case an exception in the connect to server method is raised
def test_login(self):
    self.connection = connect_to_server(user, password)

# Fail by using an assert
def test_login(self):
    self.connection = connect_to_server(user, password)
    assert self.connection.is_connected(), "We are not connected"

# Fail by raising an exception if we are not connected
def test_login(self):
    self.connection = connect_to_server(user, password)
    if not self.connection.is_connected():
        raise Exception("We are not connected")

# Fail by using the build-in fail method
def test_login(self):
    self.connection = connect_to_server(user, password)
    if not self.connection.is_connected():
        self.fail("We are not connected")

# Preferred way: fail by using the build-in fail_if method
def test_login(self):
    self.connection = connect_to_server(user, password)
    self.fail_if(not self.connection.is_connected(), "We are not connected")

# Pass or fail by return True or False
def test_login(self):
    self.connection = connect_to_server(user, password)
    return self.connection.is_connected()

The preferred way of letting a test suit pass or fail is using the fail_if method. Usually passing or failing will depend on the result of some action (executing a function, comparing a variable). The fail_if method also has a way of controlling if the test suite should continue or should be aborted. More details in the API section of this document.

Logging messages

The test suite has a build in logger for logging messages. Log messages are stored in an internal buffer (a list with strings) and are directly written to the standard output (stdout, usually the console). Messages from the standard output and error handler (stdout and stderr), are redirected to the logger. When using print(), the output is stored in the logger. If an exception is raised, the trace message from the exception is stored in the logger. The logger can be accessed by the log attribute of the test suite:

import lily_unit_test

class MyTestSuite(lily_unit_test.TestSuite):

    def test_something(self):
        # Write a log message
        self.log.info("Start test something")

Before and after running th test suite, the logger is also available:

# Initialize the test suite
ts = MyTestSuite()

ts.log.info("This is a message before running the test suite")

ts.run()

ts.log.info("This is a message after running the test suite")

Below some examples of log messages.

def test_login(self):
    # Info message
    self.log.info("Connect to server")
    self.connection = connect_to_server(user, password)

    # Debug message
    self.log.debug("Connection status: {}".format(self.connection.is_connected())

    # Let's check the connection properties using print
    # These messages will be written automatically to the logger
    # This can be useful for a quick logging of some variables
    print("Server IP  :", self.connection.get_server_ip())
    print("Server name:", self.connection.get_server_name())

    # Insert an empty line
    self.log.empty_line()

    if not self.connection.is_connected()
        # Error message
        self.log.error("We are not connected")

    return self.connection.is_connected()

Note that logging an error message NOT automatically makes the test fail.

It is possible to get the messages from the logger:

ts = MyTestSuite()
ts.run()

# Get the log messages
messages = ts.log.get_log_messages()
# Write to file
with open("test_report.txt", "w") as fp:
    # The messages is a list, we can write the list in one time
    fp.writelines(messages)

See the logger API documentation for more details.

Classification

The test suite object has a build in classification. This can be set by the CLASSIFICATION attribute.

import lily_unit_test

class MyTestSuite(lily_unit_test.TestSuite):

    CLASSIFICATION = <value>

The values are defined in an object called Classification and can be imported from the package.

import lily_unit_test

# Regular test suite
class MyTestSuite01(lily_unit_test.TestSuite):

    # By default the value is PASS, so this is not necessary
    CLASSIFICATION = lily_unit_test.Classification.PASS


# Test suite that we expect to fail
class MyTestSuite02(lily_unit_test.TestSuite):

    # Override the default value
    CLASSIFICATION = lily_unit_test.Classification.FAIL

The default value is PASS, and is usually suitable for most test suites. This means in general there is no need to override this attribute. Setting this attribute to FAIL will make the test suite pass in case of a failure. All errors are logged as usual but the end result will be passed in case of a failure. If the test suite passes, the test suite is marked as failed.

This situation is useful when the test fails because of a known issue, and you want to accept the known issue. As long as the issue is there the test will pass. When the issue is solved, the test fails, reminding you to restore the classification attribute.

The log messages will show this:

- No classification defined:
2024-01-05 19:35:54.328 | ERROR  | Test classification is not defined: None
2024-01-05 19:35:54.328 | ERROR  | Test suite TestSuiteClassification: FAILED

- Classification set to FAIL and test suite fails because of a known issue, but is accepted
2024-01-05 19:38:17.989 | INFO   | Test suite failed, but accepted because classification is set to 'FAIL'
2024-01-05 19:38:17.989 | INFO   | Test suite TestSuiteClassification: PASSED

- Classification set to FAIL and test suite passes because of the known issue is solved
2024-01-05 19:39:46.530 | ERROR  | Test suite passed, but a failure was expected because classification is set to 'FAIL'
2024-01-05 19:39:46.530 | ERROR  | Test suite TestSuiteClassification: FAILED

Subclassing the test suite

You can create your own sub class of the test suite and use that test suite sub class for running tests. This provides a way for adding your own test functions you can use in all your test suites. An example of creating your own test suite base class is shown below:

import lily_unit_test

# First we create our own test suite base class, which is a subclass of the lily test suite
class MyTestSuiteBaseClass(lily_unit_test.TestSuite):

    # Override constructor, not needed in some cases
    # Can be needed when we need to initialize stuff before running the test suite
    def __init__(self, *args):
        # initialize the lily Test Suite with parameters
        super().__init__(*args)

        # Add our own stuff to initialize
        self.my_attribute = some_value

    # Add some methods to use in your test suites
    def calculate_something_important(self):
        # Here some amazing code where we calculate something very important.


# Use our own test suite
class MyTestSuite(MyTestSuiteBaseClass):

    def test_something(self):
        # Access the added attribute
        self.my_attribute = a_new_value
        # Do some calculations
        self.calculate_something_important()


# Run the test suite
if __name__ == "__main__":

    MyTestSuite().run()

This can help you prevent duplicate code in your tests and make your test suites more maintainable.

There is a small catch. When using the test runner, it will search for any class based on the lily test suite class. Meaning in our example, it will run two test suites: MyTestSuiteBaseClass and MyTestSuite. We cannot know that MyTestSuiteBaseClass is not a test suite but only used as base class. To prevent running the base class, simply add it as an exclusion to the test runner:

from lily_unit_test import TestRunner

# Run test runner with the base class excluded
options = {
    "exclude_test_suites": ["MyTestSuiteBaseClass"]
}
TestRunner.run(".", options)

For more details about using the test runner, see the chapter about the test runner.

Test suite API

class lily_unit_test.TestSuite(report_path=None)

Base class for all test suites.

Parameters:

report_path – path were the reports are stored.

The test runner creates the report path and passes it to the test suite. This path can be used in the tests. Setting this path here will not change the path where the reports are stored. This is determined by the test runner (see test runner class).

fail(error_message, raise_exception=True)

Make the test suite fail.

Parameters:

• error_message – the error message that should be written to the logger.

• raise_exception – if True, an exception is raised and the test suite will stop.

The fail method logs an error message and raises an exception. When the exception is raised, the test suite stops and is reported as failed. Setting the raise_exception to False, does not raise an exception and the test suite continues. Even though the test suite continues it is reported as failed.

import lily_unit_test

class MyTestSuite(lily_unit_test.TestSuite):

    def test_something(self):

        # do some things

        # In case something is wrong, and we cannot continue.
        if not check_something_that_must_be_good():
            # Log a failure with exception, this will make the test suite fail and stop.
            self.fail("Something is wrong, and we cannot continue")

        # In case something is wrong, and we still can continue.
        if not check_if_something_is_ok():
            # Log a failure without exception, this will make the test suite fail.
            self.fail("Something is not OK, but we continue", False)

        # do some other stuff

fail_if(expression, error_message, raise_exception=True)

Fail if the given expression evaluates to True.

Parameters:

• expression – the expression that should be evaluated.

• error_message – the error message that should be written to the logger.

• raise_exception – if True, an exception is raised and the test suite will stop.

Same as fail() but evaluates an expression first. If the expression evaluates to True, the fail() method is executed with the given parameters.

class MyTestSuite(lily_unit_test.TestSuite):

    def test_something(self):

        # do some things

        self.fail_if(not check_something_that_must_be_good(),
                     "Something is wrong, and we cannot continue")

        self.fail_if(not check_if_something_is_ok(),
                     "Something is not OK, but we continue", False)

        # do some other stuff

get_report_path()

Get the path to the report files as set by the test runner.

Returns:

string containing the path to the report files.

run(log_traceback=False)

Run the test suite.

Parameters:

log_traceback – if True, detailed traceback information is written to the logger in case of an exception.

Returns:

True when all tests are passed, False when one or more tests are failed.

The run method creates a list of all methods starting with test_. Before executing the test methods, it executes the setup method. After executing the test methods, it executes the teardown method.

setup()

The setup method. This can be overridden in the test suite. This will be executed before running all test methods.

Returns:

True or None when the setup is passed, False when the setup is failed.

The test methods are executed after the setup is executed successfully. If the setup fails because of either an exception or returning False, the test methods are not executed.

static sleep(sleep_time)

Simple wrapper for time.sleep()

Parameters:

sleep_time – time to sleep in seconds (can be fractional)

static start_thread(target, args=())

Starts a function (target) in a separate thread with the given arguments.

Parameters:

• target – function to start as a thread

• args – tuple with arguments to pass to the thread

Returns:

a reference to the started thread

The thread is started as a daemon thread, meaning that the thread will be terminated when test execution stops. The thread can be monitored by the is_alive() method of the thread.

import lily_unit_test

class MyTestSuite(liy_unit_test.TestSuite):

    def back_ground_job(self, some_parameter):
        # do some time-consuming stuff in the background

    def test_something(self):
        # Start our background job
        t = self.start_thread(self.back_ground_job, (parameter_value, ))

        # do some other stuff while the job is running

        # Check if our job is running
        if t.is_alive():
            self.log.debug("The job is still running")

        # Wait for the job to finish, with timeout of 30 seconds, check every second.
        if self.wait_for(t.is_alive, False, 30, 1):
            self.log.debug("The job is done")
        else:
            self.fail("The thread did not finish within 30 seconds.")

        # Check result from the thread

Note that if an exception is raised in the thread, the thread is ended. The test suite will report a failure.

Note that the thread may be hanging for some reason and does not stop. When checking if the thread is finished, a timeout should be included.

teardown()

The teardown method. This can be overridden in the test suite. This will be executed after running all test methods.

This method is always executed and if there is an exception raised in this method, the test suite is reported as failed.

static wait_for(object_to_check, expected_result, timeout, interval)

Wait for a certain result with a certain timeout

Parameters:

• object_to_check – object must be a list with one element or a function. In case of a function the function is called in every iteration.

• expected_result – the expected value for the object to check.

• timeout – how long to check (float in seconds).

• interval – at what interval to check (float in seconds).

Returns:

True when the expected result is met, False when the timer times out.

This function only works with mutable variables or objects that can be called. It does not work on immutable variables since they are not passed as reference.

import lily_unit_test

class MyTestSuite(lily_unit_test.TestSuite):

    def test_wait_for_variable(self):
        # Set initial value of the variable, put in a list, so it is mutable
        self._test_value[0] = False

        # Wait for the variable to change. Wait for automatically checks the first
        # element of the list
        result = self.wait_for(self._test_value, True, 1, 0.1)

    def test_wait_for_function(self):
        # Check the outcome of a function, e.g.: checking if a server is connected.
        # Note the missing '()' for the function, we pass a reference of the function.
        result = self.wait_for(server.is_connected, True, 5, 0.1)