SDK Development


Discuss and develop the mbed SDK.

Mbed SDK automated test suite

Mbed SDK automated test suite

Introduction

Test suit allows users to run locally on their machines Mbed SDK’s tests included in Mbed SDK repository. It also allows users to create their own tests and for example add new tests to test set as they progress with their project. If test is generic enough it could be included into official Mbed SDK test pool just do it via normal pull request!

Each test is supervised by python script called “host test” which will at least Test suite is using build script API to compile and build test source together with required by test libraries like CMSIS, Mbed, Ethernet, USB etc.

Prerequisites

CompilerMbed SDK AbbreviationExample Version
Keil ARM CompilerARM, uARMARM C/C++ Compiler, 5.03 [Build 117]
GCC ARMGCC_ARMgcc version 4.8.3 20131129 (release) [ARM/embedded-4_8-branch revision 205641] (GNU Tools for ARM Embedded Processors)
GCC CodeSourceryGCC_CSgcc version 4.8.1 (Sourcery CodeBench Lite 2013.11-24)
GCC CodeRedGCC_CRgcc version 4.6.2 20121016 (release) [ARM/embedded-4_6-branch revision 192487] (GNU Tools for ARM Embedded Processors)
IAR Embedded WorkbenchIARIAR ANSI C/C++ Compiler V6.70.1.5641/W32 for ARM

Test suite core: singletest.py script

singletest.py is a test suite script which allows users to compile, build tests and test runners (also supports CppUTest UT library). Script also is responsible for test execution on devices selected by configuration files.

Parameters of singletest.py

Test execution script singletest.py is a powerful tool to run tests for Mbed SDK platform. It is flexible and allows users to configure test execution process and define which mbed platforms will be tested.

By specifying external configuration files (in JSON format) you can gain flexibility and prepare many different test scenarios. Just pass configuration file names to your script and run it.

MUTs Specification

You can easily configure your MUTs (Mbed Under Test) by creating configuration file with MUTs' description.

Note: This configuration file must be in JSON format (http://www.w3schools.com/json/).

Note: Unfortunately JSON format is not allowing you to have comments inside JSON code.

Example 1 - three MUTs configuration (no peripherals)

Lets see some example and let's try to configure small "test farm" with three devices connected to your host computer. In this example no peripherals (like SD card or EEPROM) are connected to our Mbed boards. We will use three platforms in this example:

After connecting boards to our host machine (PC) we can check which serial ports and disks they occupy. For our example let's assume that:

  • LPC1768 is on COM4 and disk drive is J:.
  • KL25Z is on COM39 and disk drive is E:.
  • NUCLEO_F103RB is on COM11 and disk drive is I:.

If you are working under Linux your port and disk could look like /dev/ttyACM5 and /media/usb5.

This information is needed to create muts_all.json configuration file. Its name will be passed to singletest.py script after -M (MUTs specification) switch. let's see how this file's content would look like in our example below:

muts_all.json

{
    "1" : {"mcu": "LPC1768",
        "port":"COM4",
        "disk":"J:\\",
        "peripherals": []
    },

    "2" : {"mcu": "KL25Z",
        "port":"COM39",
        "disk":"E:\\",
        "peripherals": []
    },

    "3" : {"mcu": "NUCLEO_F103RB",
        "port":"COM11",
        "disk":"I:\\",
        "peripherals": []
    }
}

Note: We will leave field "peripherals" empty for the sake of this example. We will explain it later. All you need to do now is to properly fill fields "mcu", "port" and "disk".

Note: Please make sure files muts_all.json and test_spec.json are in workspace_tools/ directory. We will assume in this example they are.

Where to find "mcu" names? You can use option -S of build script build.py to check all supported off-line MCUs' names. Calling build.py script with -S option in workspace_tools/ directory should give you below list:

build.py -S

~\workspace_tools> build.py -S
+---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+
| Platform      |    ARM    |    uARM   |  GCC_ARM  |   GCC_CS  |   GCC_CR  |    IAR    | GCC_CW_EWL | GCC_CW_NEWLIB |
+---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+
| ARCH_BLE      |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| ARCH_PRO      |  Default  | Supported | Supported | Supported | Supported | Supported |     -      |       -       |
| ARM_MPS2      |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| DISCO_F051R8  |     -     |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| DISCO_F100RB  |     -     |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| DISCO_F303VC  |     -     |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| DISCO_F407VG  |     -     |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| HRM1017       |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| K20D5M        |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| K64F          |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| KL05Z         | Supported |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| KL25Z         |  Default  |     -     | Supported |     -     |     -     |     -     | Supported  |   Supported   |
| KL46Z         |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| LPC1114       | Supported |  Default  | Supported |     -     | Supported |     -     |     -      |       -       |
| LPC11C24      |  Default  | Supported | Supported |     -     |     -     |     -     |     -      |       -       |
| LPC11U24      | Supported |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| LPC11U24_301  |  Default  | Supported | Supported |     -     |     -     |     -     |     -      |       -       |
| LPC11U35_401  | Supported |  Default  | Supported |     -     | Supported |     -     |     -      |       -       |
| LPC11U35_501  | Supported |  Default  | Supported |     -     | Supported |     -     |     -      |       -       |
| LPC11U68      |     -     |  Default  | Supported |     -     | Supported |     -     |     -      |       -       |
| LPC1347       |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| LPC1549       |     -     |  Default  |     -     |     -     | Supported |     -     |     -      |       -       |
| LPC1768       |  Default  | Supported | Supported | Supported | Supported | Supported |     -      |       -       |
| LPC2368       |  Default  |     -     | Supported |     -     | Supported |     -     |     -      |       -       |
| LPC4088       |  Default  |     -     | Supported |     -     | Supported |     -     |     -      |       -       |
| LPC4330_M4    |  Default  |     -     | Supported |     -     | Supported | Supported |     -      |       -       |
| LPC810        |     -     |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| LPC812        |     -     |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| LPCCAPPUCCINO |     -     |  Default  | Supported |     -     | Supported |     -     |     -      |       -       |
| NRF51822      |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_F030R8 | Supported |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_F072RB | Supported |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_F103RB | Supported |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_F302R8 | Supported |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_F334R8 | Supported |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_F401RE | Supported |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_F411RE | Supported |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_L053R8 | Supported |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| NUCLEO_L152RE | Supported |  Default  |     -     |     -     |     -     |     -     |     -      |       -       |
| STM32F3XX     | Supported |  Default  | Supported |     -     |     -     |     -     |     -      |       -       |
| STM32F407     |  Default  |     -     | Supported |     -     |     -     |     -     |     -      |       -       |
| UBLOX_C027    |  Default  | Supported | Supported | Supported | Supported | Supported |     -      |       -       |
| XADOW_M0      | Supported |  Default  | Supported |     -     | Supported |     -     |     -      |       -       |
+---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+
*Default - default on-line compiler
*Supported - supported off-line compiler

Total permutations: 119

Note: If you update devices' firmware or even disconnect / reconnect Mbed device you may find that serial port / disk configuration changed. You need to update configuration file accordingly or you will face connection problems and obviously tests will not run.


Now we've already defined how our devices are connected to our host PC. We can continue and define which of this MUTs will be tested and which compilers we will use to compile and build Mbed SDK and tests. To do so we need to create test specification file (let's call it test_spec.json) and put inside our configuration file information about which MUTs we actually want to test. We will pass this file's name to singletest.py script using -i switch.

Below we can see how sample test_spec.json file content could look like. (I've included all possible toolchains, we will change it in a moment).

test_spec.json

{
    "targets": {
        "LPC1768" : ["ARM", "uARM", "GCC_ARM", "GCC_CS", "GCC_CR", "IAR"],
        "KL25Z" : ["ARM", "GCC_ARM"],
        "NUCLEO_F103RB" : ["ARM", "uARM"]
     }
}

Above example configuration will force tests for LPC1768, KL25Z, NUCLEO_F103RB platforms and:

  • Compilers: ARM, uARM, GCC_ARM, GCC_CS, GCC_CR and IAR will be used to compile tests for LPC1768.
  • Compilers: ARM and GCC_ARM will be used for KL25Z platform.
  • Compilers: ARM and uARM will be used for NUCLEO_F103RB platform.

For our example purposes let's assume we only have Keil ARM compiler, so let's change configuration in test_spec.json file and reduce number of compiler to those we actually have:

test_spec.json

{
    "targets": {
        "LPC1768" : ["ARM", "uARM"],
        "KL25Z" : ["ARM"],
        "NUCLEO_F103RB" : ["ARM", "uARM"]
     }
}

Example 1 - Configuration summary

We've successfully connected all parts of our small "test farm" in two files test_spec.json and muts_all.json. Let's see how configuration logic looks like in Example 1:

/media/uploads/PrzemekWirkus/example_singletest_config.png

Note: uARM is flavor of ARM compiler. In case of uARM we will link with microlib so we can save some space on your Mbed flash. There are other limitations concerning using of microlib (e.g. it does not support C++) but we will not discuss them here further.

Just a reminder. If you are unsure which Mbed platform supports which toolchains please go to workspace_tools/ directory in Mbed SDK repository and call build.py script with option -S:

c:\temp\repos\mbed\workspace_tools> python build.py -S
+---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+
| Platform      |    ARM    |    uARM   |  GCC_ARM  |   GCC_CS  |   GCC_CR  |    IAR    | GCC_CW_EWL | GCC_CW_NEWLIB |
+---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+
...
| KL25Z         | Supported |     -     | Supported |     -     |     -     |     -     | Supported  |   Supported   |
...
| LPC1768       | Supported | Supported | Supported | Supported | Supported | Supported |     -      |       -       |
...
| NUCLEO_F103RB | Supported | Supported | Supported |     -     |     -     |     -     |     -      |       -       |
...
+---------------+-----------+-----------+-----------+-----------+-----------+-----------+------------+---------------+
Total permutations: 101

Example 1 - Run your tests

After you configure all your MUTs and compilers you are ready to run tests. Make sure your devices are connected and your configuration files reflect your current configuration (serial ports, devices). Go to workspace_tools directory in your mbed location.

cd workspace_tools/

and execute test script.

singletest.py -i test_spec.json -M muts_all.json

Example 1 - summary

Having multiple configuration files allows you to manage your test scenarios in more flexible manner. You can:

  • set up all platforms and toolchains used during testing,
  • define (using script's -n switch) which tests you want to run during testing or
  • just run regression (all tests). Regression is default setting for test script.

You can also force singletest.py script to:

  • run only peripherals' tests (switch -P) or
  • just skip peripherals' tests (switch -C).

Below you can see full singletes.py help command (-h switch) listing:

c:\temp\repos\mbed\workspace_tools> singletest.py -h
Usage: singletest.py [options]

This script allows you to run mbed defined test cases for particular MCU(s)
and corresponding toolchain(s).

Options:
  -h, --help            show this help message and exit
  -i FILE, --tests=FILE
                        Points to file with test specification
  -M FILE, --MUTS=FILE  Points to file with MUTs specification (overwrites
                        settings.py and private_settings.py)
  -j NUMBER, --jobs=NUMBER
                        Define number of compilation jobs. Default value is 1
  -g, --goanna-for-tests
                        Run Goanna static analyse tool for tests. (Project
                        will be rebuilded)
  -G, --goanna-for-sdk  Run Goanna static analyse tool for mbed SDK (Project
                        will be rebuilded)
  -s, --suppress-summary
                        Suppresses display of wellformatted table with test
                        results
  -t, --test-summary    Displays wellformatted table with test x toolchain
                        test result per target
  -r, --test-automation-report
                        Prints information about all tests and exits
  -R, --test-case-report
                        Prints information about all test cases and exits
  -P, --only-peripherals
                        Test only peripheral declared for MUT and skip common
                        tests
  -C, --only-commons    Test only board internals. Skip perpherials tests and
                        perform common tests.
  -c COPY_METHOD, --copy-method=COPY_METHOD
                        You can choose which copy method you want to use put
                        bin in mbed. You can choose from 'cp', 'copy',
                        'xcopy'. Default is python shutils.copy method.
  -n TEST_BY_NAMES, --test-by-names=TEST_BY_NAMES
                        Runs only test enumerated it this switch
  -S, --supported-toolchains
                        Displays supported matrix of MCUs and toolchains
  -O, --only-build      Only build tests, skips actual test procedures
                        (flashing etc.)
  --config              Displays full test specification and MUTs configration
                        and exits
  --loops=TEST_LOOPS_LIST
                        Set no. of loops per test. Format:
                        TEST_1=1,TEST_2=2,TEST_3=3
  --global-loops=TEST_GLOBAL_LOOPS_VALUE
                        Set global number of test loops per test. Default
                        value is set 1
  --firmware-name=FIRMWARE_GLOBAL_NAME
                        Set global name for all produced projects. Note,
                        proper file extension will be added by buid scripts.
  -u, --shuffle         Shuffles test execution order
  --shuffle-seed=SHUFFLE_TEST_SEED
                        Shuffle seed (If you want to reproduce your shuffle
                        order please use seed provided in test summary)
  --reset-type=MUT_RESET_TYPE
                        Extra reset method used to reset MUT by host test
                        script
  -f GENERAL_FILTER_REGEX, --filter=GENERAL_FILTER_REGEX
                        For some commands you can use filter to filter out
                        results
  --inc-timeout=NUMBER  You can increase global timeout for each test by
                        specifying additional test timeout in seconds
  -l LOG_FILE_NAME, --log=LOG_FILE_NAME
                        Log events to external file (note not all console
                        entries may be visible in log file)
  --verbose-skipped     Prints some extra information about skipped tests
  -V, --verbose-test-result
                        Prints test serial output
  -v, --verbose         Verbose mode (prints some extra information)
  --version             Prints script version and exits

Example: singletest.py -i test_spec.json -M muts_all.json

Example 2

This example will show you how to configure single device, run general tests or only peripheral tests. We will also show some real test result examples.

  1. We will test only one board ST Nucleo board F334R8 connected to our PC (port COM 46 and disk is E:).
  2. We will also connect EEPROM 24LC256 to SDA, SCL pins of our Nucleo board and define 24LC256 peripheral to make sure our test suite will run all available tests for 24LC256.
  • Let's configure our one MUT and set uARM as the only compiler we will use to compiler Mbed SDK and tests.
  • We also need to create two configuration files muts_all.json and test_spec.json to pass our small testbed configuration to test script.

muts_all.json

{
    "1" : { 
            "mcu": "NUCLEO_F334R8",
            "port":"COM46",
            "disk":"E:\\",
            "peripherals": ["24LC256"]
    }
}

Note: By defining "peripherals": ["24LC256"] we are passing to test suite information that this particular board has EEPROM 24LC256 connected to our board.

test_spec.json

{
    "targets": {
        "NUCLEO_F334R8" : ["uARM"]
     }
}

Above configuration now looks like this:

/media/uploads/PrzemekWirkus/example_2_config.png

  • Please make sure device is connected before we will start running tests.
  • Please make sure files muts_all.json and test_spec.json are in workspace_tools/ directory. We will assume in this example they are.

To build Mbed SDK, compile, build and run tests please execute below command in workspace_tools/ directory.

singletest.py -i test_spec.json -M muts_all.json

~\workspace_tools> singletest.py -i test_spec.json -M muts_all.json

>>> BUILD LIBRARY CMSIS (NUCLEO_F334R8, uARM)

>>> BUILD LIBRARY MBED (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: I2C_EEPROM_LINE (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_A25::I2C EEPROM line read/write test [OK] in 12.41 of 15 sec

>>> BUILD PROJECT: BASIC (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_A1::Basic [OK] in 3.42 of 10 sec

>>> BUILD PROJECT: DEV_NULL (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::EXAMPLE_1::/dev/null [OK] in 3.42 of 10 sec

>>> BUILD PROJECT: TIMEOUT (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_24::Timeout Int us [OK] in 11.47 of 15 sec

>>> BUILD PROJECT: TIME_US (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_25::Time us [OK] in 11.43 of 15 sec

>>> BUILD PROJECT: DIV (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_26::Integer constant division [OK] in 3.37 of 10 sec

>>> BUILD PROJECT: TICKER_2 (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_23::Ticker Int us [OK] in 12.43 of 15 sec

>>> BUILD PROJECT: I2C_EEPROM (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_A19::I2C EEPROM read/write test [OK] in 11.42 of 15 sec

>>> BUILD PROJECT: TICKER (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_11::Ticker Int [OK] in 12.43 of 20 sec

>>> BUILD PROJECT: HELLO (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_10::Hello World [OK] in 2.42 of 10 sec

>>> BUILD PROJECT: CPP (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_12::C++ [OK] in 3.42 of 10 sec

>>> BUILD PROJECT: RTC (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_16::RTC [OK] in 4.76 of 15 sec

>>> BUILD PROJECT: STDIO (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_2::stdio [UNDEF] in 20.42 of 20 sec

>>> BUILD PROJECT: ECHO (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::uARM::MBED_A9::Serial Echo at 115200 [UNDEF] in 10.37 of 10 sec
Test summary:
+--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
| Result | Target        | Toolchain | Test ID   | Test Description                | Elapsed Time (sec) | Timeout (sec) |
+--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
| OK     | NUCLEO_F334R8 | uARM      | MBED_A25  | I2C EEPROM line read/write test |       12.41        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_A1   | Basic                           |        3.42        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | EXAMPLE_1 | /dev/null                       |        3.42        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_24   | Timeout Int us                  |       11.47        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_25   | Time us                         |       11.43        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_26   | Integer constant division       |        3.37        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_23   | Ticker Int us                   |       12.43        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_A19  | I2C EEPROM read/write test      |       11.42        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_11   | Ticker Int                      |       12.43        |       20      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_10   | Hello World                     |        2.42        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_12   | C++                             |        3.42        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_16   | RTC                             |        4.76        |       15      |
| UNDEF  | NUCLEO_F334R8 | uARM      | MBED_2    | stdio                           |       20.42        |       20      |
| UNDEF  | NUCLEO_F334R8 | uARM      | MBED_A9   | Serial Echo at 115200           |       10.37        |       10      |
+--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
Result: 2 UNDEF / 12 OK

Completed in 160 sec

If we want to get additional test summary with results in separate columns please use option -t.

singletest.py -i test_spec.json -M muts_all.json -t

Test summary:
+---------------+-----------+---------------------------------+-------+
| Target        | Test ID   | Test Description                | uARM  |
+---------------+-----------+---------------------------------+-------+
| NUCLEO_F334R8 | EXAMPLE_1 | /dev/null                       | OK    |
| NUCLEO_F334R8 | MBED_10   | Hello World                     | OK    |
| NUCLEO_F334R8 | MBED_11   | Ticker Int                      | OK    |
| NUCLEO_F334R8 | MBED_12   | C++                             | OK    |
| NUCLEO_F334R8 | MBED_16   | RTC                             | OK    |
| NUCLEO_F334R8 | MBED_2    | stdio                           | UNDEF |
| NUCLEO_F334R8 | MBED_23   | Ticker Int us                   | OK    |
| NUCLEO_F334R8 | MBED_24   | Timeout Int us                  | OK    |
| NUCLEO_F334R8 | MBED_25   | Time us                         | OK    |
| NUCLEO_F334R8 | MBED_26   | Integer constant division       | OK    |
| NUCLEO_F334R8 | MBED_A1   | Basic                           | OK    |
| NUCLEO_F334R8 | MBED_A19  | I2C EEPROM read/write test      | OK    |
| NUCLEO_F334R8 | MBED_A25  | I2C EEPROM line read/write test | OK    |
| NUCLEO_F334R8 | MBED_A9   | Serial Echo at 115200           | UNDEF |
+---------------+-----------+---------------------------------+-------+

To only compile, build and link all required libraries and tests' dependencies please use option -O (capital 'o').

singletest.py -i test_spec.json -M muts_all.json -O

workspace_tools> singletest.py -i test_spec.json -M muts_all.json -O

>>> BUILD LIBRARY CMSIS (NUCLEO_F334R8, uARM)

>>> BUILD LIBRARY MBED (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: I2C_EEPROM_LINE (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: BASIC (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: DEV_NULL (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: TIMEOUT (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: TIME_US (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: DIV (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: TICKER_2 (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: I2C_EEPROM (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: TICKER (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: HELLO (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: CPP (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: RTC (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: STDIO (NUCLEO_F334R8, uARM)

>>> BUILD PROJECT: ECHO (NUCLEO_F334R8, uARM)
Completed in 1 sec

To add extra verbose prints please use option -v.

singletest.py -i test_spec.json -M muts_all.json -v

~\workspace_tools> singletest.py -i test_spec.json -M muts_all.json -v
{   u'targets': {   u'NUCLEO_F334R8': [u'uARM']}}
{   u'1': {   u'disk': u'E:\\',
              u'mcu': u'NUCLEO_F334R8',
              u'peripherals': [u'24LC256'],
              u'port': u'COM46'}}

>>> BUILD LIBRARY CMSIS (NUCLEO_F334R8, uARM)

>>> BUILD LIBRARY MBED (NUCLEO_F334R8, uARM)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)
TargetTest::NUCLEO_F334R8::TestSkipped(ethernet)

>>> BUILD PROJECT: I2C_EEPROM_LINE (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
I2C: I2C Frequency: 400000 Hz
I2C: Lines pattern write test ... [OK]
I2C: Write errors: 0 ... [OK]
I2C: Lines pattern read test ... [OK]
I2C: Read errors: 0/1000 ... [OK]
EEPROM: Pattern match errors: 0/1000 ... [OK]
{{success}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_A25::I2C EEPROM line read/write test [OK] in 12.40 of 15 sec

>>> BUILD PROJECT: BASIC (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
{{success}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_A1::Basic [OK] in 3.42 of 10 sec

>>> BUILD PROJECT: DEV_NULL (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Received 33 bytes

{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::EXAMPLE_1::/dev/null [OK] in 3.42 of 10 sec
TargetTest::NUCLEO_F334R8::TestSkipped(digital_loop)
TargetTest::NUCLEO_F334R8::TestSkipped(TMP102)
TargetTest::NUCLEO_F334R8::TestSkipped(digital_loop)

>>> BUILD PROJECT: TIMEOUT (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Test started
. skipped
. skipped
. skipped
. in 1.01 sec (0.01) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.01 sec (0.01) [OK]
. in 1.01 sec (0.01) [OK]
. in 1.01 sec (0.01) [OK]
. in 1.01 sec (0.01) [OK]
. in 1.01 sec (0.01) [OK]
Completed in 9.08 sec

{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_24::Timeout Int us [OK] in 11.54 of 15 sec

>>> BUILD PROJECT: TIME_US (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Test started
. skipped
. skipped
. skipped
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
Completed in 9.01 sec

{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_25::Time us [OK] in 11.48 of 15 sec

>>> BUILD PROJECT: DIV (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
64bit: 0x7FFFFFFF8: expected 0x55555555 got 0x55555555 ... [OK]
64bit: 0x17FFFFFFE8: expected 0xFFFFFFFF got 0xFFFFFFFF ... [OK]
{{success}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_26::Integer constant division [OK] in 3.43 of 10 sec

>>> BUILD PROJECT: TICKER_2 (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Test started
. skipped
. skipped
. skipped
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
Completed in 9.00 sec

{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_23::Ticker Int us [OK] in 11.45 of 15 sec

>>> BUILD PROJECT: I2C_EEPROM (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
I2C: I2C Frequency: 400000 Hz
I2C: Write 0x66 at address 0x0000 test ...
Unable to write data to EEPROM (i2c_stat = 0x01), aborting
{{failure}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_A19::I2C EEPROM read/write test [FAIL] in 3.43 of 15 sec
TargetTest::NUCLEO_F334R8::TestSkipped(port_loop)
TargetTest::NUCLEO_F334R8::TestSkipped(port_loop)
TargetTest::NUCLEO_F334R8::TestSkipped(SD)

>>> BUILD PROJECT: TICKER (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Test started
. skipped
. skipped
. skipped
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
. in 1.00 sec (0.00) [OK]
Completed in 9.00 sec

{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_11::Ticker Int [OK] in 11.43 of 20 sec

>>> BUILD PROJECT: HELLO (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Hello World

{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_10::Hello World [OK] in 2.43 of 10 sec

>>> BUILD PROJECT: CPP (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Static::init
Static::stack_test
Stack::init
Stack::hello
Stack::destroy
Static::check_init: OK
Heap::init
Heap::hello
Heap::check_init: OK
Heap::destroy
{{success}}
{{end}}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_12::C++ [OK] in 3.43 of 10 sec

>>> BUILD PROJECT: RTC (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Got RTC time: [1256729737] [2009-10-28 11:35:37 AM] ... OK
Got RTC time: [1256729738] [2009-10-28 11:35:38 AM] ... OK
Got RTC time: [1256729739] [2009-10-28 11:35:39 AM] ... OK
Got RTC time: [1256729740] [2009-10-28 11:35:40 AM] ... OK
Got RTC time: [1256729741] [2009-10-28 11:35:41 AM] ... OK

{success}
{end}
Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_16::RTC [OK] in 4.72 of 15 sec

>>> BUILD PROJECT: STDIO (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Generated number: 3112

Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_2::stdio [UNDEF] in 20.38 of 20 sec
TargetTest::NUCLEO_F334R8::TestSkipped(digital_loop)

>>> BUILD PROJECT: ECHO (NUCLEO_F334R8, uARM)
Test::Output::Start
Mbed: "COM46" "E:\"
Starting the ECHO test

Test::Output::Finish
TargetTest::NUCLEO_F334R8::uARM::MBED_A9::Serial Echo at 115200 [UNDEF] in 10.43 of 10 sec
Test summary:
+--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
| Result | Target        | Toolchain | Test ID   | Test Description                | Elapsed Time (sec) | Timeout (sec) |
+--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
| OK     | NUCLEO_F334R8 | uARM      | MBED_A25  | I2C EEPROM line read/write test |        12.4        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_A1   | Basic                           |        3.42        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | EXAMPLE_1 | /dev/null                       |        3.42        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_24   | Timeout Int us                  |       11.54        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_25   | Time us                         |       11.48        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_26   | Integer constant division       |        3.43        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_23   | Ticker Int us                   |       11.45        |       15      |
| FAIL   | NUCLEO_F334R8 | uARM      | MBED_A19  | I2C EEPROM read/write test      |        3.43        |       15      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_11   | Ticker Int                      |       11.43        |       20      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_10   | Hello World                     |        2.43        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_12   | C++                             |        3.43        |       10      |
| OK     | NUCLEO_F334R8 | uARM      | MBED_16   | RTC                             |        4.72        |       15      |
| UNDEF  | NUCLEO_F334R8 | uARM      | MBED_2    | stdio                           |       20.38        |       20      |
| UNDEF  | NUCLEO_F334R8 | uARM      | MBED_A9   | Serial Echo at 115200           |       10.43        |       10      |
+--------+---------------+-----------+-----------+---------------------------------+--------------------+---------------+
Result: 1 FAIL / 2 UNDEF / 11 OK

Completed in 150 sec

Run only selected tests

You can select which tests should be executed when you run test suite. Use -n switch to define tests by their ids you want to execute. Use comma to separate test ids:

singletest.py -i test_spec.json -M muts_all.json -n RTOS_1,RTOS_2,RTOS_3,MBED_10,MBED_16,MBED_11

Below you can see test summary only for selected tests:

Test summary:
+--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+
| Result | Target  | Toolchain | Test ID | Test Description        | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+
| OK     | LPC1768 | ARM       | MBED_11 | Ticker Int              |       12.51        |       20      |  1/1  |
| OK     | LPC1768 | ARM       | MBED_10 | Hello World             |        2.42        |       10      |  1/1  |
| OK     | LPC1768 | ARM       | MBED_16 | RTC                     |        5.67        |       15      |  1/1  |
| OK     | LPC1768 | ARM       | RTOS_3  | Semaphore resource lock |        11.5        |       20      |  1/1  |
| OK     | LPC1768 | ARM       | RTOS_2  | Mutex resource lock     |       12.51        |       20      |  1/1  |
| OK     | LPC1768 | ARM       | RTOS_1  | Basic thread            |       12.58        |       15      |  1/1  |
+--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+
Result: 6 OK

Completed in 75.23 sec

Shuffle your tests

We strongly encourage you to shuffle your test order each time you execute test suite script. Rationale: It is probable that tests executed in one particular order will pass and in other will fail.

To shuffle your tests’ order please use option –u (or --shuffle):

singletest.py -i test_spec.json -M muts_all.json --shuffle

Above command with force test script to randomly generate shuffle seed and shuffle test order execution. Note: Shuffle seed is float in [0.0, 1.0).

You can always recreate particular test order by forcing shuffle (-u or --shuffle} switch) and passing shuffle seed back to test suite using --shuffle-seed switch:

singletest.py -i test_spec.json -M muts_all.json --shuffle --shuffle-seed 0.4041028336

Note: You can also supply your own randomly generated shuffle seed to drive particular test execution order scenarios. Just make sure shuffle seed is float in [0.0, 1.0).

You can find test shuffle seed in test summary:

...
| OK     | LPC1768 | ARM       | RTOS_6    | Mail messaging              |        3.42        |       10      |  1/1  |
| OK     | LPC1768 | ARM       | RTOS_4    | Signals messaging           |        7.45        |       10      |  1/1  |
| OK     | LPC1768 | ARM       | MBED_A9   | Serial Echo at 115200       |        2.84        |       10      |  1/1  |
+--------+---------+-----------+-----------+-----------------------------+--------------------+---------------+-------+
Result: 1 FAIL / 22 OK
Shuffle Seed: 0.4041028336

Completed in 234.85 sec

Note: you can force shuffle seed (--shuffle-seed) will only be available when you use shuffle switch -u. In other case test suite will execute tests in one fixed order.

Loop over your tests

You can loop (execute few times) tests while test script is running.

  • Use switch --global-loops to define global loop count or

singletest.py -i test_spec.json -M muts_all.json --global-loops 5

Example output, each test is repeated 5 times:

Building library CMSIS (LPC1768, ARM)
Building library MBED (LPC1768, ARM)
Building project TICKER (LPC1768, ARM)
TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.48 of 20 sec
TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.51 of 20 sec
TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.53 of 20 sec
TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.47 of 20 sec
TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.47 of 20 sec
Building project HELLO (LPC1768, ARM)
TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.42 of 10 sec
TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.37 of 10 sec
TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.37 of 10 sec
TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.37 of 10 sec
TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.37 of 10 sec
Building project RTC (LPC1768, ARM)
TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.59 of 15 sec
TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.67 of 15 sec
TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.66 of 15 sec
TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.67 of 15 sec
TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.67 of 15 sec
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project SEMAPHORE (LPC1768, ARM)
TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.51 of 20 sec
TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.49 of 20 sec
TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.48 of 20 sec
TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.49 of 20 sec
TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.50 of 20 sec
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project MUTEX (LPC1768, ARM)
TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.50 of 20 sec
TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.50 of 20 sec
TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.50 of 20 sec
TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.51 of 20 sec
TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.50 of 20 sec
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project BASIC (LPC1768, ARM)
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.51 of 15 sec
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.52 of 15 sec
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.58 of 15 sec
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.51 of 15 sec
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.57 of 15 sec
Test summary:
+--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+
| Result | Target  | Toolchain | Test ID | Test Description        | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+
| OK     | LPC1768 | ARM       | MBED_11 | Ticker Int              |       12.47        |       20      |  5/5  |
| OK     | LPC1768 | ARM       | MBED_10 | Hello World             |        2.37        |       10      |  5/5  |
| OK     | LPC1768 | ARM       | MBED_16 | RTC                     |        5.67        |       15      |  5/5  |
| OK     | LPC1768 | ARM       | RTOS_3  | Semaphore resource lock |        11.5        |       20      |  5/5  |
| OK     | LPC1768 | ARM       | RTOS_2  | Mutex resource lock     |        12.5        |       20      |  5/5  |
| OK     | LPC1768 | ARM       | RTOS_1  | Basic thread            |       12.57        |       15      |  5/5  |
+--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+
Result: 6 OK

Completed in 374.42 sec

Note: Default global loop value is 1.

  • switch --loops to overwrite global loop count and redefine loop count for particular test(s). Define test loops as TEST_ID=X where X is integer and separate loops count definitions by comma if necessary. E.g. TEST_1=5,TEST_2=20,TEST_3=2.

singletest.py -i test_spec.json -M muts_all.json RTOS_1=10,RTOS_2=5

Example output, where test RTOS_1 will be executed 10 times and test RTOS_2 will be executed 5 times:

Building library CMSIS (LPC1768, ARM)
Building library MBED (LPC1768, ARM)
Building project TICKER (LPC1768, ARM)
TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.52 of 20 sec
Building project HELLO (LPC1768, ARM)
TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.42 of 10 sec
Building project RTC (LPC1768, ARM)
TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.63 of 15 sec
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project SEMAPHORE (LPC1768, ARM)
TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.50 of 20 sec
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project MUTEX (LPC1768, ARM)
TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.52 of 20 sec
TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.49 of 20 sec
TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.50 of 20 sec
TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.49 of 20 sec
TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.50 of 20 sec
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project BASIC (LPC1768, ARM)
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.52 of 15 sec
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.48 of 15 sec
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.63 of 15 sec
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.54 of 15 sec
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.52 of 15 sec
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.58 of 15 sec
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.51 of 15 sec
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.58 of 15 sec
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.58 of 15 sec
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.58 of 15 sec
Test summary:
+--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+
| Result | Target  | Toolchain | Test ID | Test Description        | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+
| OK     | LPC1768 | ARM       | MBED_11 | Ticker Int              |       12.52        |       20      |  1/1  |
| OK     | LPC1768 | ARM       | MBED_10 | Hello World             |        2.42        |       10      |  1/1  |
| OK     | LPC1768 | ARM       | MBED_16 | RTC                     |        5.63        |       15      |  1/1  |
| OK     | LPC1768 | ARM       | RTOS_3  | Semaphore resource lock |        11.5        |       20      |  1/1  |
| OK     | LPC1768 | ARM       | RTOS_2  | Mutex resource lock     |        12.5        |       20      |  5/5  |
| OK     | LPC1768 | ARM       | RTOS_1  | Basic thread            |       12.58        |       15      | 10/10 |
+--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+
Result: 6 OK

Completed in 278.60 sec

Example 3

Please do not forget you can combine few options together to get result you want. For example you want to repeat few tests multiple number of times, shuffle test ids execution order and select only tests which are critical for you at this point. You can do it using switch -n, --global-loops with --loops and --shuffle:

Execute above command to:

  • run only tests: RTOS_1, RTOS_2, RTOS_3, MBED_10, MBED_16, MBED_11.
  • shuffle test execution order. Note tests in loops will not be shuffled.
  • set global loop count to 3 - each test will repeated 3 times.
  • overwrite global loop count (set above to 3) and:
    • force to loop test RTOS_1 to execute 3 times.
    • force to loop test RTOS_2 to execute 4 times.
    • force to loop test RTOS_3 to execute 5 times.
    • force to loop test MBED_11 to execute 5 times.

Note: In test summary you can find shuffle seed value for this test script execution: 0.8837216077.

singletest.py -i test_spec.json -M muts_all.json -n RTOS_1,RTOS_2,RTOS_3,MBED_10,MBED_16,MBED_11 --shuffle --global-loops 3 --loops RTOS_1=3,RTOS_2=4,RTOS_3=5,MBED_11=5

Example output:

Building library CMSIS (LPC1768, ARM)
Building library MBED (LPC1768, ARM)
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project BASIC (LPC1768, ARM)
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.51 of 15 sec
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.52 of 15 sec
TargetTest::LPC1768::ARM::RTOS_1::Basic thread [OK] in 12.57 of 15 sec
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project SEMAPHORE (LPC1768, ARM)
TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.49 of 20 sec
TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.49 of 20 sec
TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.49 of 20 sec
TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.50 of 20 sec
TargetTest::LPC1768::ARM::RTOS_3::Semaphore resource lock [OK] in 11.49 of 20 sec
Building project TICKER (LPC1768, ARM)
TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.47 of 20 sec
TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.51 of 20 sec
TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.47 of 20 sec
TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.42 of 20 sec
TargetTest::LPC1768::ARM::MBED_11::Ticker Int [OK] in 12.51 of 20 sec
Building project HELLO (LPC1768, ARM)
TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.37 of 10 sec
TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.42 of 10 sec
TargetTest::LPC1768::ARM::MBED_10::Hello World [OK] in 2.37 of 10 sec
Building project RTC (LPC1768, ARM)
TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.64 of 15 sec
TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.66 of 15 sec
TargetTest::LPC1768::ARM::MBED_16::RTC [OK] in 5.67 of 15 sec
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project MUTEX (LPC1768, ARM)
TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.51 of 20 sec
TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.50 of 20 sec
TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.54 of 20 sec
TargetTest::LPC1768::ARM::RTOS_2::Mutex resource lock [OK] in 12.51 of 20 sec
Test summary:
+--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+
| Result | Target  | Toolchain | Test ID | Test Description        | Elapsed Time (sec) | Timeout (sec) | Loops |
+--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+
| OK     | LPC1768 | ARM       | RTOS_1  | Basic thread            |       12.57        |       15      |  3/3  |
| OK     | LPC1768 | ARM       | RTOS_3  | Semaphore resource lock |       11.49        |       20      |  5/5  |
| OK     | LPC1768 | ARM       | MBED_11 | Ticker Int              |       12.51        |       20      |  5/5  |
| OK     | LPC1768 | ARM       | MBED_10 | Hello World             |        2.37        |       10      |  3/3  |
| OK     | LPC1768 | ARM       | MBED_16 | RTC                     |        5.67        |       15      |  3/3  |
| OK     | LPC1768 | ARM       | RTOS_2  | Mutex resource lock     |       12.51        |       20      |  4/4  |
+--------+---------+-----------+---------+-------------------------+--------------------+---------------+-------+
Result: 6 OK
Shuffle Seed: 0.8837216077

Completed in 300.14 sec

What is host test?

Test suite support test supervisor concept. This concept is realized by separate Python script called "host test". Host tests can be found in workspace_tools/host_tests/ directory. Host test script is executed in parallel with test runner to monitor test execution. Basic host test just monitors device's default serial port for test results returned by test runner. Simple tests will print test result on serial port. In other cases host tests can for example judge by test results returned by test runner is test passed or failed. It all depends on test itself. In some cases host test can be TCP server echoing packets from test runner and judging packet loss. In other cases it can just check if values returned from accelerometer are actually valid (sane).

Peripherals testing

When creating MUTs configuration file (switch -M) you can define in MUTs JSON file peripherals connected to your device:

muts_all.json

{
    "1" : {"mcu" : "KL25Z",
           "port" : "COM39",
           "disk" : "E:\\",
           "peripherals" : ["SD", "24LC256"]}
}

You can force test suite to run only common tests (switch -C) or only peripheral tests (switch -P).

singletest.py -i test_spec.json -M muts_all.json -C

will not include tests for SD card and EEPROM 24LC256.

singletest.py -i test_spec.json -M muts_all.json -P

will only run tests bind to SD card and EEPROM 24LC256.

Note: option -P is useful for example in cases when you have same platform and different shields you want to test. No need to test common part all the time (timers, RTC, RTOS etc.). You can force to test peripherals only on some devices and for example only common tests on other devices.

Additional MUTs configuration file settings

You can add extra information to each MUT configuration. In particular you can specify which flashing (binary copy method) should be used, how to reset target and for example set reset timeout (used to delay test execution just after reset).

muts_all.json

{
    "1" : {"mcu" : "LPC1768",
           "port" : "COM77",
           "disk" : "G:\\",
           "peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop", "SD"]},

    "2" : {"mcu" : "KL25Z",
           "port" : "COM89",
           "disk" : "F:\\",
           "peripherals" : ["SD", "24LC256", "KL25Z"],
           "copy_method" : "copy",
           "reset_type" : "default",
           "reset_tout" : "2"},

    "3" : {"mcu" : "LPC11U24",
           "port" : "COM76",
           "disk" : "E:\\",
           "peripherals" : []}
}

Please note that for MUT no. 2 few extra parameters were defined: copy_method, reset_type and reset_tout. Using this extra options you can tell test suite more about MUT you are using. This will allow you to be more flexible in terms of how you configure and use your MUTs.

  • copy_method - STRING - tells test suite which binary copy method should be used.
  • reset_type - STRING - some boards may require special reset handling, for example vendor specific command must be executed to reset device.
  • reset_tout - INTEGER - extra timeout just after device is reseted. May be used to wait for few seconds so device may finish booting, flashing data internally etc.

You may notice that singletest.py command line help contains description about: - Option -c (in MUTs file called copy_method) with available copy methods supported by test suite plugin system. - Option -r (in MUTs file called reset_type) with available reset methods supported by test suite plugin system.

Part of help listing for singletest.py:

  -c COPY_METHOD, --copy-method=COPY_METHOD
                        Select binary copy (flash) method. Default is Python's
                        shutil.copy() method. Plugin support: copy, cp,
                        default, eACommander, eACommander-usb, xcopy
  -r MUT_RESET_TYPE, --reset-type=MUT_RESET_TYPE
                        Extra reset method used to reset MUT by host test
                        script. Plugin support: default, eACommander,
                        eACommander-usb

Set common output binary name for all tests

In some cases you would like to have the same name for all tests. You can use switch {--firmware-name to specify (without extension) build script output binary name.

In below example we would like to have all test binaries called firmware.bin (or with other extension like .elf, .hex depending on target accepted format).

singletest.py -i test_spec.json -M muts_all.json --firmware-name firmware

Building library CMSIS (LPC1768, ARM)
Building library MBED (LPC1768, ARM)
Building project TICKER to firmware (LPC1768, ARM)
Building project HELLO to firmware (LPC1768, ARM)
Building project RTC to firmware (LPC1768, ARM)
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project SEMAPHORE to firmware (LPC1768, ARM)
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project MUTEX to firmware (LPC1768, ARM)
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project BASIC to firmware (LPC1768, ARM)
Completed in 0.39 sec

Only build libraries and tests

You can force test script to only build all libraries and tests:

singletest.py -i test_spec.json -M muts_all.json -O

Building library CMSIS (LPC1768, ARM)
Building library MBED (LPC1768, ARM)
Building project TICKER (LPC1768, ARM)
Building project HELLO (LPC1768, ARM)
Building project RTC (LPC1768, ARM)
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project SEMAPHORE (LPC1768, ARM)
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project MUTEX (LPC1768, ARM)
Building library RTX (LPC1768, ARM)
Building library RTOS (LPC1768, ARM)
Building project BASIC (LPC1768, ARM)
Completed in 0.37 sec

Note: this can be useful when you are integrating test script with CI (Continuous Integration system). You can make libraries and test building process as separate CI job step. And test execution as another separate step which will be not executed by CI system if e.g. building tests with dependencies fails.

Build your libraries and tests on multiple CPUs

Use option -j to build tests with dependencies on multiple cores.

Below example will use 4 (four) cores to build your dependencies to tests:

singletest.py -i test_spec.json -M muts_all.json -j 4

Check MUTs and test specification configuration

You can check MUTs configuration read from configuration file (switch -M) and test specification read from test specification file (switch -i) by setting switch ---config.

Below command will check configuration against configuration in test_spec.json and muts_all.json file.

singletest.py -i test_spec.json -M muts_all.json --config

MUTs configuration in muts_all.json:
+-------+-------------+---------------+------+-------+
| index | peripherals | mcu           | disk | port  |
+-------+-------------+---------------+------+-------+
| 13    | 24LC256     | NUCLEO_L152RE | F:\  | COM9  |
| 15    | 24LC256     | NUCLEO_F401RE | G:\  | COM25 |
| 14    | 24LC256     | NUCLEO_F302R8 | E:\  | COM34 |
| 17    | 24LC256     | NUCLEO_L053R8 | E:\  | COM35 |
| 16    | 24LC256     | NUCLEO_F030R8 | H:\  | COM12 |
| 4     | 24LC256     | NUCLEO_F103RB | I:\  | COM11 |
+-------+-------------+---------------+------+-------+

Test specification in test_spec.json:
+---------------+-----+------+
| mcu           | ARM | uARM |
+---------------+-----+------+
| NUCLEO_F103RB | Yes | Yes  |
| NUCLEO_F030R8 | Yes | Yes  |
| NUCLEO_L152RE | Yes | Yes  |
| NUCLEO_F401RE | Yes | Yes  |
| NUCLEO_F302R8 | Yes | Yes  |
+---------------+-----+------+

Simple checks can be performed to see if you actually properly configured test specification: In below example user defined his/hers test specification file like this:

test_spec.json

{
    "targets": {
        "NUCLEO_F103RB": ["uARM", "ARM", "IAR"],
        "NUCLEO_F030R8": ["uARM", "ARM"],
        "NUCLEO_F401RE": ["uARM", "ARM"],
        "NUCLEO_F302R8": ["uARM", "ARM", "GCC_ARM"],
        "NUCLEO_L152RE": ["uARM", "ARM"]
    }
}

Because NUCLEO platform do not support IAR or GCC_ARM toolchains "Toolchain conflicts" will be printed like below:

MUTs configuration in muts_all.json:
+-------+-------------+---------------+------+-------+
| index | peripherals | mcu           | disk | port  |
+-------+-------------+---------------+------+-------+
| 13    | 24LC256     | NUCLEO_L152RE | F:\  | COM9  |
| 15    | 24LC256     | NUCLEO_F401RE | G:\  | COM25 |
| 14    | 24LC256     | NUCLEO_F302R8 | E:\  | COM34 |
| 17    | 24LC256     | NUCLEO_L053R8 | E:\  | COM35 |
| 16    | 24LC256     | NUCLEO_F030R8 | H:\  | COM12 |
| 4     | 24LC256     | NUCLEO_F103RB | I:\  | COM11 |
+-------+-------------+---------------+------+-------+

Test specification in test_spec.json:
+---------------+-----+---------+------+------+
| mcu           | ARM | GCC_ARM | IAR  | uARM |
+---------------+-----+---------+------+------+
| NUCLEO_F103RB | Yes | -       | Yes* | Yes  |
| NUCLEO_F030R8 | Yes | -       | -    | Yes  |
| NUCLEO_L152RE | Yes | -       | -    | Yes  |
| NUCLEO_F401RE | Yes | -       | -    | Yes  |
| NUCLEO_F302R8 | Yes | Yes*    | -    | Yes  |
+---------------+-----+---------+------+------+
Toolchain conflicts:
        * Target NUCLEO_F103RB does not support IAR toolchain
        * Target NUCLEO_F302R8 does not support GCC_ARM toolchain

JSON configuration file parsing hints

Test suite script can supply you with simple configuration file JSON related error notifications. For example when you have a typo in your JSON configuration file:

test_spec.json

{
    "targets": {
        "NUCLEO_F103RB": ["uARM"],
        "NUCLEO_F030R8": ["uARM"],
        "NUCLEO_F401RE": ["uARM"],
        "NUCLEO_F302R8": ["uARM"] typo!,
        "NUCLEO_L152RE": ["uARM"]
    }
}

You should be notified and pointed to place where correct JSON parsing was interrupted:

singletest.py -i test_spec.json -M muts_all.json

Results in:

JSON file test_spec.json parsing failed. Reason: Expecting , delimiter: line 6 column 35 (char 158)

Line 1: {
Line 2:     "targets": {
Line 3:         "NUCLEO_F103RB": ["uARM"],
Line 4:         "NUCLEO_F030R8": ["uARM"],
Line 5:         "NUCLEO_F401RE": ["uARM"],
Line 6:         "NUCLEO_F302R8": ["uARM"] typo!,
        ----------------------------------^

Vocabulary

Test runner

A test runner is an executable program that runs tests implemented using test framework and reports the test results.

MUT

MUT stands for Mbed Under Test - a device we have had connected to our host computer and

Appendix A

Example muts_all.json file

muts_all.json

{
    "1" : {"mcu" : "LPC1768",
           "port" : "COM4",
           "disk" : "J:\\",
           "peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop", "SD"]},

    "2" : {"mcu" : "KL25Z",
           "port" : "COM39",
           "disk" : "E:\\",
           "peripherals" : ["SD", "24LC256", "KL25Z"]},

    "3" : {"mcu" : "LPC11U24",
           "port" : "COM5",
           "disk" : "E:\\",
           "peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop"]},

    "4" : {"mcu" : "NUCLEO_F103RB",
           "port" : "COM11",
           "disk" : "I:\\",
           "peripherals" : ["24LC256"]},

    "5" : {"mcu" : "NRF51822",
           "port" : "COM21",
           "disk" : "I:\\",
           "peripherals" : []},

    "6" : {"mcu" : "UBLOX_C027",
           "port" : "COM14",
           "disk" : "F:\\",
           "peripherals" : []},

    "7" : {"mcu" : "LPC2368",
           "port" : "COM15",
           "disk" : "G:\\",
           "peripherals" : ["TMP102", "digital_loop", "port_loop", "analog_loop", "SD"]},

    "8" : {"mcu" : "LPC812",
           "port" : "COM6",
           "disk" : "E:\\",
           "peripherals" : ["24LC256"]},

    "9" : {"mcu" : "LPC1549",
           "port" : "COM32",
           "disk" : "I:\\",
           "peripherals" : ["24LC256"]},

    "10" : {"mcu" : "LPC4088",
            "port" : "COM17",
            "disk" : "E:\\",
            "peripherals" : []},

    "11" : {"mcu" : "KL46Z",
            "port" : "COM40",
            "disk" : "F:\\",
            "peripherals" : ["SD", "24LC256", "KL46Z"]},

    "12" : {"mcu" : "K64F",
            "port" : "COM18",
            "disk" : "E:\\",
            "peripherals" : ["ethernet"]},

    "13" : {"mcu" : "NUCLEO_L152RE",
            "port" : "COM9",
            "disk" : "E:\\",
            "peripherals" : []},

    "14" : {"mcu" : "NUCLEO_F302R8",
            "port" : "COM34",
            "disk" : "E:\\",
            "peripherals" : ["24LC256"]},

    "15" : {"mcu" : "NUCLEO_F401RE",
            "port" : "COM47",
            "disk" : "E:\\",
            "peripherals" : ["SD"]},

    "16" : {"mcu" : "NUCLEO_F030R8",
            "port" : "COM12",
            "disk" : "H:\\",
            "peripherals" : ["24LC256"]},

    "17" : {"mcu" : "NUCLEO_L053R8",
            "port" : "COM35",
            "disk" : "E:\\",
            "peripherals" : ["24LC256"]},

    "18" : {"mcu" : "LPC11U68",
            "port" : "COM36",
            "disk" : "E:\\",
            "peripherals" : ["24LC256"]},

    "19" : {"mcu" : "NUCLEO_F411RE",
            "port" : "COM44",
            "disk" : "F:\\",
            "peripherals" : ["24LC256"]},

    "20" : {"mcu" : "NUCLEO_F072RB",
            "port" : "COM45",
            "disk" : "F:\\",
            "peripherals" : ["24LC256"]},

    "21" : {"mcu" : "NUCLEO_F334R8",
            "port" : "COM46",
            "disk" : "E:\\",
            "peripherals" : ["24LC256"]},

    "22" : {"mcu" : "XADOW_M0",
            "port" : "COM51",
            "disk" : "E:\\",
            "peripherals" : []}
}

Example test_spec.json file

test_spec.json

{
    "targets": {
        "KL25Z": ["ARM", "GCC_ARM"],
        "LPC1768": ["ARM", "GCC_ARM", "GCC_CR", "GCC_CS", "IAR"],
        "LPC11U24": ["uARM"],
        "NRF51822": ["ARM"],
        "LPC2368": ["ARM"],
        "LPC4088": ["ARM"],
        "NUCLEO_F334R8" : ["uARM"]
    }
}