18th, March

Data Driven Testing with Qt

!! Code available on my GitHub toskyRocker account !!

Hi everyone,
today I’m here to show a brief example of a completely Data Driven Test Suite implemented with the Qt Framework.

The main goals are:

• Showing how to execute data driven tests with QtTest.
• Handling input and verification data directly from a configuration file.

The focus will be on the configuration and execution of the test rather than on the complexity of the test itself.
Before digging into the details, clone the repository and open the main project in QtCreator by clicking on Compoent_A_UnitTest_Project.pro.

The software has been developed with Qt 5.8.0 and QtCreator 4.4.1 in Windows 10.

Configure then the Build and Run Settings in QtCreator as it follows:

QtCreator Build Settings

As depicted in the picture I specified the shadow build, providing its path and I added an extra Build Step where I added: “install clean” in Make Arguments.

This is needed when you want to perform the install step in order to have all the dependencies installed in your bin folder.

QtCreator Run Settings

In the Run Settings shown above I simply specified the configuration file needed to run the test.

The configuration file can be found at the following path of this repo: Qt_Data_Driven_Test_Suite\Component_A_UnitTest\component_a_unit_test_data.json.

To execute the program then, just specify the following command line argument:
-config C:\Users\[path-to-repo]\Qt_Data_Driven_Test_Suite\Component_A_UnitTest\component_a_unit_test_data.json

Now that we have configured the project we are ready to go through the details.
The following UML diagrams show the structural and behavioural views of the Test.

Let’s say we have a class named Component_A which will be our software under test.
In this simple example Component_A has got only the following method doPow()  which implements the exponentiation provided
a base and an exponent values as input.

Component_A_UnitTest will have an instance of Component_A and will run the tests on it by executing doPow() as many times as the entries of the Test Suite.
In order to avoid hardcoded and make the same program reusable with different inputs, the configuration file has been conceived to store all input and verification data belonging to each Test Case defined in the Test Suite.

From the picture below, you can see that the configuration file consists of a json file where a list of Test Cases are defined.
For each Test Case, an ID, input data and verification data are provided so that they can be loaded up and used in the test execution.

Each block, representing a Test Case holds the following information:

• Test Case ID: identifier
• Base value: used as input for the exponentation
• Exponent value: used as input for the exponentation
• Expected result: value used in the verification step to check whether the implementation of the exponentiation works as expected.

Component A Unit Test Configuration File

In addition to Component_A, the diagram below depicts all the other classes in the project that contribute to run the tests:
• Component_A_UnitTest is the principal class, implementing the Qt data driven strategy. It takes advantage of an instance of
  JsonFileParser to store the data read from the configuration file.
• JsonFileParser is an helper class which provides the functionalities for reading the configuration file and storing the extracted information in member data structures (list of TestCaseInfo objects).
• TestCaseInfo is the class which holds all the information belonging to a single Test Case specified in the configuration file.

Component A UnitTest Class Diagram

The execution of a single Test happens in testProcedurePow(), where providing the right input values,
Component_A_UnitTest calls Component_A::doPow(), fetches the result and compares is with the expected one.

As described on Qt Documentation (Qt Test Tutorial),
for each test procedure X(), the user has to define its matching X_data() method which is foundamental for loading and making available all the data needed in each test procedure execution.
X_data() builds its internal representation of the Test Suite where each column defines a parameter and each row represents a test.
The Qt application will then execute X() as many times as the number of rows defined in its table.

In our specific case, since we have only 2 Test Cases described in the configuration file, there will be 2 execution of testProcedurePow(), each time of course providing and validating the right information, thanks to the mechanism explained.

The following snippet show exactly this strategy where in testProcedurePow_data() I populate the table by adding each row’s information.

Each testProcedurePow() needs to know the information previously defined in the configuration file and imported thanks to JsonFileParser.
This is performed in testProcedurePow_data() which populates the table with all the information.

 
void Component_A_UnitTest::testProcedurePow()
{
    // Fetching all the needed data to run the test from the table
    // defined in testProcedure_pow_data()
    // Fetching the information by providing the key
    QFETCH(QString, test_case_id);
    QFETCH(QString, base);
    QFETCH(QString, exp);
    QFETCH(QString, expected_result);

    qDebug() << "Component_A_UnitTest::testProcedurePow - " << test_case_id;

    // Converting the input data to the needed type
    int baseValue = base.toInt();
    int exponentValue = exp.toInt();
    int expectedValue = expected_result.toInt();

    // Performing the needed operation that need to be tested
    int actualValue = testData->cmpA->doPow(baseValue, exponentValue);

    // Verificaiton Step
    QCOMPARE(actualValue, expectedValue);

}

void Component_A_UnitTest::testProcedurePow_data()
{
    // Populating Columns' headers
    QTest::addColumn<QString>(TEST_CASE_ID);
    QTest::addColumn<QString>(BASE);
    QTest::addColumn<QString>(EXP);
    QTest::addColumn<QString>(EXPECTED_RESULT);

    // Populating each row i.e. test
    for(TestCaseInfo testCaseInfo : testData->parser->getTestCaseInfoList())
    {
        // Doesn't accept QString. It needs to be converted to std String
        QString testCaseID = testCaseInfo.getTestCaseID();
        const char* testCaseID_std = testCaseID.toStdString().c_str();
        QString base = testCaseInfo.getInputData().value(BASE);
        QString exp = testCaseInfo.getInputData().value(EXP);
        QString expectedResult = testCaseInfo.getVerificationData().value(EXPECTED_RESULT);

        QTest::newRow(testCaseID_std)
                << testCaseID
                << base
                << exp
                << expectedResult;
    }
}

A behavioural overview is shown in the sequence diagram below that remarks the data driven approach implemented by Qt.
The core of the program relies in the loop that iterates over the rows of the table, calling each time and sequentially: init(), testProcedurePow(), cleanup().

Component A UnitTest Sequence Diagram