libostree and $OSTREE_REPO environment path

libostree is a great tool to handle incremental or full updates for an Linux filesystem.
But virtually all commands of ostree requires the –repo argument to override the default system repository path. This is really annoying after a while so my first attempt to get rid of this was to create an alias

alias ost='ostree --repo=/tmp/repo'

It works but is not good. To solve it once and for all I was about to implement support for getting the repo path from an environment variable.

When digging around in the code, I found that the support is already there but completely undocumented.

It is therefor already possible to

export OSTREE_REPO=/tmp/repo
ostree <cmd>

So the patch for this time is simply a line that mention the option to use $OSTREE_REPO environment variable to point out a directory as system repository path.

commit 075e676eb63a3abaf647c789e22d8d1afe3a1dd5
Author: Marcus Folkesson <marcus.folkesson@gmail.com>
Date:   Tue Oct 17 21:03:23 2017 +0200

docs: mention the $OSTREE_REPO environment variable

$OSTREE_REPO may be set to override the default location
of the repository.

Link: https://mail.gnome.org/archives/ostree-list/2017-October/msg00003.html

Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>

Closes: #1282
Approved by: cgwalters

Conclusion

If /ostree/repo is not the location of your system repository, it is really preferable to set the $OSTREE_REPO environment variable instead of provide the –repo to every single command.

FIT vs legacy image format

U-Boot supports several image formats when booting a kernel. However, a Linux system usually need multiple files for booting.
Such files may be the kernel itself, an initrd and a device tree blob.

A typical embedded Linux system have all these files in at least two-three different configurations. It is not uncommon to have

  • Default configuration
  • Rescue configuration
  • Development configuration
  • Production configuration

Only these four configurations may involve twelve different files. Or maybe the devicetree is shared between two configurations. Or maybe the initrd is… Or maybe the….. you got the point. It has a farily good chance to end up quite messy.

This is a problem with the old legacy formats and is somehow addressed in the FIT (Flattened Image Tree) format.

Lets first look at the old formats before we get into FIT.

zImage

The most well known format for the Linux kernel is the zImage. The zImage contains a small header followed by a self-extracting code and finally the payload itself.
U-Boot support this format by the bootz command.

Image layout

zImage format
Header
Decompressing code
Compressed Data

uImage

When talking about U-Boot, the zImage is usually encapsulated in a file called uImage created with the mkimage utility. Beside image data, the uImage also contains information such as OS type, loader information, compression type and so on.
Both data and header is checksumed with CRC32.

The uImage format also supports multiple images. The zImage, initrd and devicetree blob may therefor be included in one single monolithic uImage.
Drawbacks with this monolith is that it has no flexible indexing, no hash integrity and no support for security at all.

Image layout

uImage format
Header
Header checksum
Data size
Data load address
Entry point address
Data CRC
OS, CPU
Image type
Compression type
Image name
Image data

FIT

The FIT (Flattened Image Tree) has been around for a while but is for unknown reason rarely used in real systems, based on my experience. FIT is a tree structure like Device Tree (before they change format to YAML: https://lwn.net/Articles/730217/) and handles images of various types.

These images is then used in something called configurations. One image may be a used in several configurations.

This format has many benifits compared to the other:

  • Better solutions for multi component images
    • Multiple kernels (productions, feature, debug, rescue…)
    • Multiple devicetrees
    • Multiple initrds
  • Better hash integrity of images. Supports different hash algorithms like
    • SHA1
    • SHA256
    • MD5
  • Support signed images
    • Only boot verified images
    • Detect malware

Image Tree Source

The file describing the structure is called .its (Image Tree Source).
Here is an example of a .its file

/dts-v1/;

/ {
    description = "Marcus FIT test";
    #address-cells = <1>;

    images {
        kernel@1 {
            description = "My default kenel";
            data = /incbin/("./zImage");
            type = "kernel";
            arch = "arm";
            os = "linux";
            compression = "none";
            load = <0x83800000>;
            entry = <0x83800000>;
            hash@1 {
                algo = "md5";
            };
        };

        kernel@2 {
            description = "Rescue image";
            data = /incbin/("./zImage");
            type = "kernel";
            arch = "arm";
            os = "linux";
            compression = "none";
            load = <0x83800000>;
            entry = <0x83800000>;
            hash@1 {
                algo = "crc32";
            };
        };

        fdt@1 {
            description = "FDT for my cool board";
            data = /incbin/("./devicetree.dtb");
            type = "flat_dt";
            arch = "arm";
            compression = "none";
            hash@1 {
                algo = "crc32";
            };
        };


    };

    configurations {
        default = "config@1";

        config@1 {
            description = "Default configuration";
            kernel = "kernel@1";
            fdt = "fdt@1";
        };

        config@2 {
            description = "Rescue configuration";
            kernel = "kernel@2";
            fdt = "fdt@1";
        };

    };
};

This .its file has two kernel images (default and rescue) and one FDT image.
Note that the default kernel is hashed with md5 and the other with crc32. It is also possible to use several hash-functions per image.

It also specifies two configuration, Default and Rescue. The two configurations is using different kernels (well, it is the same kernel in this case since both is pointing to ./zImage..) but share the same FDT. It is easy to build up new configurations on demand.

The .its file is then passed to mkimage to generate an itb (Image Tree Blob):

mkimage -f kernel.its kernel.itb

which is bootable from U-Boot.

Look at ./doc/uImage.FIT/ in the U-Boot source code for more examples on how a .its-file can look like.
It also contains examples on signed images which is worth a look.

Boot from U-Boot

To boot from U-Boot, use the bootm command and specify the physical address for the FIT image and the configuration you want to boot.

Example on booting config@1 from the .its above:

bootm 0x80800000#config@1

Full example with bootlog:

U-Boot 2015.04 (Oct 05 2017 - 14:25:09)

CPU:   Freescale i.MX6UL rev1.0 at 396 MHz
CPU:   Temperature 42 C
Reset cause: POR
Board: Marcus Cool board
fuse address == 21bc400
serialnr-low : e1fe012a
serialnr-high : 243211d4
       Watchdog enabled
I2C:   ready
DRAM:  512 MiB
Using default environment

In:    serial
Out:   serial
Err:   serial
Net:   FEC1
Boot from USB for mfgtools
Use default environment for                              mfgtools
Run bootcmd_mfg: run mfgtool_args;bootz ${loadaddr} - ${fdt_addr};
Hit any key to stop autoboot:  0
=> bootm 0x80800000#config@1
## Loading kernel from FIT Image at 80800000 ...
   Using 'config@1' configuration
   Trying 'kernel@1' kernel subimage
     Description:  My cool kernel
     Type:         Kernel Image
     Compression:  uncompressed
     Data Start:   0x808000d8
     Data Size:    7175328 Bytes = 6.8 MiB
     Architecture: ARM
     OS:           Linux
     Load Address: 0x83800000
     Entry Point:  0x83800000
     Hash algo:    crc32
     Hash value:   f236d022
   Verifying Hash Integrity ... crc32+ OK
## Loading fdt from FIT Image at 80800000 ...
   Using 'config@1' configuration
   Trying 'fdt@1' fdt subimage
     Description:  FDT for my cool board
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x80ed7e4c
     Data Size:    27122 Bytes = 26.5 KiB
     Architecture: ARM
     Hash algo:    crc32
     Hash value:   1837f127
   Verifying Hash Integrity ... crc32+ OK
   Booting using the fdt blob at 0x80ed7e4c
   Loading Kernel Image ... OK
   Loading Device Tree to 9ef86000, end 9ef8f9f1 ... OK

Starting kernel ...

One thing to think about

Since this FIT image tends to grow in size, it is a good idea to set CONFIG_SYS_BOOTM_LEN in the U-Boot configuration.

- CONFIG_SYS_BOOTM_LEN:
        Normally compressed uImages are limited to an
        uncompressed size of 8 MBytes. If this is not enough,
        you can define CONFIG_SYS_BOOTM_LEN in your board config file
        to adjust this setting to your needs.

Conclusion

I advocate to use the FIT format because it solves many problems that can get really awkward. Just the bonus to program one image in production instead of many is a great benifit. The configurations is well defined and there is no chance that you will end up booting one kernel image with an incompatible devicetree or whatever.

Use FIT images should be part of the bring-up-board activity, not the last thing todo before closing a project. The "Use seperate kernel/dts/initrds for now and then move on to FIT when the platform is stable" does not work. It simple not gonna happend, and it is a mistake.

config utility for Buildroot

I’m using the ./scripts/config script in the Linux kernel tree a lot. The script is used to manipulate a .config file from the command line which is quite nice to be able to do.

I use it mostly to enable configurations from a script or as a part of automated tests.

Buildroot is also using KBuild as its configuration system so I adapted this script and submitted a patch. It is now part of the mainline.

commit f2c5c7a263d9d61faba80544e0f6ac6da03716ee
Author: Marcus Folkesson <marcus.folkesson@gmail.com>
Date:   Tue Sep 26 23:15:02 2017 +0200

utils/config: new script to manipulate .config files on the command line

Default prefix is set to `BR2_` but may be overidden by setting
BR2_PREFIX.

Example usage:

Enable `BR2_PACKAGE_GNUPG`:
./utils/config --package --enable GNUPG

Check state of config option `BR2_PACKAGE_GNUPG`:
./utils/config --package --state GNUPG
y

Enable `BR2_PACKAGE_GNUPG`:
./utils/config --package --disable GNUPG

Set `BR2_TARGET_GENERIC_ISSUE` to "Welcome":
./utils/config --set-str  TARGET_GENERIC_ISSUE "Welcome"

Copied from the Linux kernel (4.13-rc6) source code and adapted to
Buildroot.
Thanks to Andi Kleen who is the original author of this script.

Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>
[Arnout: merge the two tr invications]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

In addition to all commands found in he Linux kernel version, I also addes the –package config as a shortcut to operate on packages.