Test packages in Buildroot

When writing packages for Buildroot there are several conditions that you have to test your package against.

This includes different toolchains, architectures, C-libraries, thread-implementations and more. To help you with that, Buildroot provides the utils/test-pkg script.

Nothing describes the script better than its own help text [1]:

test-pkg: test-build a package against various toolchains and architectures

The supplied config snippet is appended to each toolchain config, the
resulting configuration is checked to ensure it still contains all options
specified in the snippet; if any is missing, the build is skipped, on the
assumption that the package under test requires a toolchain or architecture
feature that is missing.

In case failures are noticed, you can fix the package and just re-run the
same command again; it will re-run the test where it failed. If you did
specify a package (with -p), the package build dir will be removed first.

The list of toolchains is retrieved from support/config-fragments/autobuild/toolchain-configs.csv.
Only the external toolchains are tried, because building a Buildroot toolchain
would take too long. An alternative toolchains CSV file can be specified with
the -t option. This file should have lines consisting of the path to the
toolchain config fragment and the required host architecture, separated by a
comma. The config fragments should contain only the toolchain and architecture
settings.

By default, a useful subset of toolchains is tested. If needed, all
toolchains can be tested (-a), an arbitrary number of toolchains (-n
in order, -r for random).

Hands on

In these examples I'm going to build the CRIU package that i recently worked on.

First I will create a config snippet that contains the necessary options to enable my packet. In my case it's CRIU and HOST_PYTHON3:

     cat > criu.config <<EOF
     BR2_PACKAGE_HOST_PYTHON3=y
     BR2_PACKAGE_CRIU=y
     EOF

I can now start test-pkg and provide the config snippet:

     utils/test-pkg -c criu.config -p criu
                     bootlin-armv5-uclibc [1/6]: FAILED
                      bootlin-armv7-glibc [2/6]: FAILED
                    bootlin-armv7m-uclibc [3/6]: SKIPPED
                      bootlin-x86-64-musl [4/6]: OK
                       br-arm-full-static [5/6]: SKIPPED
                             sourcery-arm [6/6]: FAILED
     6 builds, 2 skipped, 3 build failed, 0 legal-info failed, 0
show-info failed

Some of them fails, the end of the build log is available at ~/br-test-pkg/*/logfile.

Now start read the logfiles and fix the errors for each failed test.

Once the test-pkg (without -a) will have no failures, it will be a good test to retry it with the -a (run all tests) option:

     utils/test-pkg -a -c criu.config -p criu

As soon as more and more tests passes, it takes unnecessary amount of time to run the same (successful) test again, so it's better to hand-pick those tests that actually fails.

To do this you may provide a specific list of toolchain configurations:

     cp support/config-fragments/autobuild/toolchain-configs.csv
criu-toolchains.csv
     # edit criu-toolchains.csv to keep your toolchains of interest.
     utils/test-pkg -a -t criu-toolchains.csv -c criu.config -p criu

This will retest only the toolchains kept in the csv.