Design: TestBartender

Archtictural decision for lazy test evaluation

pytest does testcollection at the beginning.

We decided to do lazy testcollection.

The test duration might differ a lot - between a few seconds and 10 minutes.
The goal is, to have the overall time of the tests as small as possible.

Influencing the test process / freedom in the choice of tentacles

There might be redundant tentacles to speed up critical path.
For tests requireing two tentacles, for example WLAN STA vs WLAN AP, we may pick tentacles which are ready just now.

Classes involved in testcollection

The TestBartender and the testcollection is quite complex.

The sources are in this package: src/testbed/testcollection The sources of the test specs are in this package: src/testbed/testrunspecs

There is some unit testing here: <testbed_micropython>/tests/test_collection.py

TestRunSpec

A TestRunSpec is used to specify a test in the micropython directory which may be called in a test run.

The method generate() will generate the next TestRun - for example a call to ‘run-perfbench.py’.

The method generate() might be seen as a bartender which hands out a burger or asks for money depending on the actual needs.

        classDiagram
    class TestRunSpec{
        generate(available_tentacles, available_firmwares) -> TestRun
    }
    TestRunSpec *-- TestRunSpecs

TESTRUNSPEC_PERFTEST = TestRunSpec("run-perfbench.py", TestRunPerfTest)
TESTRUNSPEC_RUNTESTS_MULTINET = TestRunSpec("run-multitests.py", TestRunMultitestMultinet)
TESTRUNSPEC_RUNTESTS_STANDARD = TestRunSpec("run-tests.py", TestRunRunTests)

specs = [
    multinet.TESTRUNSPEC_RUNTESTS_MULTBLUETOOTH,
    multinet.TESTRUNSPEC_RUNTESTS_MULTINET,
    perftest.TESTRUNSPEC_PERFTEST,
    runtests_net_inet.TESTRUNSPEC_RUNTESTS_NET_HOSTED,
    runtests_net_inet.TESTRUNSPEC_RUNTESTS_NET_INET,
    runtests.TESTRUNSPEC_RUNTESTS_STANDARD,
    runtests.TESTRUNSPEC_RUNTESTS_EXTMOD_HARDWARE,
]

TestRun

A TestRun will

create a directory <repo>/results/RUN-TESTS_EXTMOD_HARDWARE[5f2c-RPI_PICO_W]
start a process which
- Lock the associated tentacles
- initalizes the associated tentacles
- flashes the boards if needed
- starts the test, for example ‘run-perfbench.py’
- Release the associated tentacles

The TestRun knows the associated TestRunSpec which will provide information needed to run the test.

        classDiagram
    class TestRun{
        testid 'RUN-TESTS_EXTMOD_HARDWARE[5f2c-RPI_PICO_W]'
        assigned tentacles
        assigned firmwares
        test()
    }
    class TestRunPerfTest{
        test()
    }
    class TestRunRunTests{
        test()
    }
    class TestRunMultitestBase{
        test()
    }
    TestRun <|-- TestRunPerfTest
    TestRun <|-- TestRunRunTests
    TestRun <|-- TestRunMultitestBase

TestRunSpecs generating TestRun

        classDiagram
    class TestRunSpec{
        generate(available_tentacles, available_firmwares) -> TestRun
    }
    class TestRunSpecs{
        generate(available_tentacles, available_firmwares) -> TestRun
    }
    class TestRun{
        +testid Example: RUN-TESTS_EXTMOD_HARDWARE[5f2c-RPI_PICO_W]
        +assigned tentacles
        +assigned firmwares
        -test()
    }
    class TestRunPerfTest{
        -test()
    }
    class TestRunRunTests{
        -test()
    }
    class TestRunMultitestBase{
        -test()
    }
    TestRun <|-- TestRunPerfTest
    TestRun <|-- TestRunRunTests
    TestRun <|-- TestRunMultitestBase
    TestRun --> TestRunSpec
    TestRunSpec *-- TestRunSpecs

The TestBartender calls TestRunSpecs.generate() to get a new TestRun.

Generate will loop over all TestRunSpecs.generate().
This will be a possible long list of possible ‘TestRun’s.
Not the ‘TestRun’s are ordered by priority.
The one with the highest priority is selected.
This ‘TestRun’ now will be started - flow see above.

        sequenceDiagram
   participant m as main()
   participant f as FirmwareBartender
   participant b as Build Process
   participant s as Filesystem
   m->>f: connected tentacles
   f-->>+b: firmware-variant list
   b->>+s: RPI_PICO2
   s-->>-b: done
   b-->>f: EventFirmwareSpec(RPI_PICO2)
   b->>+s: TEENSY40
   s-->>-b: done
   b-->>f: EventFirmwareSpec(TEENSY40)
   b->>+s: RPI_PICO2-RISCV
   s-->>-b: done
   b-->>f: EventFirmwareSpec(RPI_PICO2-RISCV)
   b-->>-f: EventExitFirmware()