br2-readonly-rootfs-overlay

br2-readonly-rootfs-overlay

This is a Buildroot external module that could be used as a reference design when building your own system with an overlayed root filesystem. It's created as an external module to make it easy to adapt for your to your own application.

The goal is to achieve the same functionality I have in meta-readonly-rootfs-overlay [1] but for Buildroot.

Why does this exists?

Having a read-only root file system is useful for many scenarios:

  • Separate user specific changes from system configuration, and being able to find differences
  • Allow factory reset, by deleting the user specific changes
  • Have a fallback image in case the user specific changes made the root file system no longer bootable.

Because some data on the root file system changes on first boot or while the system is running, just mounting the complete root file system as read-only breaks many applications. There are different solutions to this problem:

  • Symlinking/Bind mounting files and directories that could potentially change while the system is running to a writable partition
  • Instead of having a read-only root files system, mounting a writable overlay root file system, that uses a read-only file system as its base and writes changed data to another writable partition.

To implement the first solution, the developer needs to analyse which file needs to change and then create symlinks for them. When doing factory reset, the developer needs to overwrite every file that is linked with the factory configuration, to avoid dangling symlinks/binds. While this is more work on the developer side, it might increase the security, because only files that are symlinked/bind-mounted can be changed.

This br2-external provides the second solution. Here no investigation of writable files are needed and factory reset can be done by just deleting all files or formatting the writable volume.

How does it work?

This external module makes use of Dracut [2] to create a rootfs.cpio that is embedded into the Linux kernel as an initramfs.

The initramfs mounts the base root filsystem as read-only and the read-write filesystem as the upper layer in an overlay filesystem structure.

Dependencies

The setup requires the following kernel configuration (added as fragment file in board/qemu-x86_64/linux.fragment):

CONFIG_OVERLAY_FS=y
CONFIG_BLK_DEV_INITRD=y

Test it out

This module will build a x86_64 target that is prepared to be emulated with qemu.

Clone this repository

git clone https://github.com/marcusfolkesson/br2-readonly-rootfs-overlay.git

Build

Run build.sh to make a full build.

The script is simple, it just update the Buildroot submodule and start a build:

#!/bin/bash

# Update submodules
git submodule init
git submodule update

# Build buildroot
cd ./buildroot
make BR2_EXTERNAL=../ readonly-rootfs-overlay_defconfig
make

Artifacts

The following artifacts are generated in ./buildroot/output/images/ :

  • bzImage - The Linux kernel
  • start-qemu.sh - script that will start qemu-system-x86_64 and emulate the whole setup
  • sdcard.img - The disk image containing two partitions, one for the read-only rootfs and one for the writable upper filesystem
  • rootfs.cpio - The initramfs file system that is embedded into the kernel
  • rootfs.ext2 - The root filesystem image
  • overlay.ext4 - Empty filsystem image used for the writable layer

Emulate in QEMU

An script to emulate the whole thing is generated in the output directory. Execute the ./buildroot/output/images/start-qemu.sh script to start the emulator.

Once the system has booted, you are able to login as root:

Welcome to Buildroot
buildroot login: root

And as you can see, the root filesystem is overlayed as it should:

$ mount
...
/dev/vda1 on /media/rfs/ro type ext2 (ro,noatime,nodiratime)
/dev/vda2 on /media/rfs/rw type ext4 (rw,noatime,errors=remount-ro)
overlay on / type overlay (rw,relatime,lowerdir=/media/rfs/ro,upperdir=/media/rfs/rw/upperdir,workdir=/media/rfs/rw/work)
...

Kernel command line parameters

These examples are not meant to be complete. They just contain parameters that are used by the initscript of this repository. Some additional paramters might be necessary.

Example without initramfs

root=/dev/sda1 rootrw=/dev/sda2 init=/init

This cmd line starts /sbin/init with /dev/sda1 partition as the read-only rootfs and the /dev/sda2 partition as the read-write persistent state. When using this init script without an initrd, init=/init has to be set.

root=/dev/sda1 rootrw=/dev/sda2 init=/init rootinit=/bin/sh

The same as before but it now starts /bin/sh instead of /sbin/init

Details

All kernel parameters that is used to configure br2-readonly-rootfs-overlay:

  • root - specifies the read-only root file system device. If this is not specified, the current rootfs is used.
  • `rootfstype if support for the read-only file system is not build into the kernel, you can specify the required module name here. It will also be used in the mount command.
  • rootoptions specifies the mount options of the read-only file system. Defaults to noatime,nodiratime.
  • rootinit if the init parameter was used to specify this init script, rootinit can be used to overwrite the default (/sbin/init).
  • rootrw specifies the read-write file system device. If this is not specified, tmpfs is used.
  • rootrwfstype if support for the read-write file system is not build into the kernel, you can specify the required module name here. It will also be used in the mount command.
  • rootrwoptions specifies the mount options of the read-write file system. Defaults to rw,noatime,mode=755.
  • rootrwreset set to yes if you want to delete all the files in the read-write file system prior to building the overlay root files system.

Test packages in Buildroot

Test packages in Buildroot

When writing packages for Buildroot there are several conditions that you have to test your package against.

This includes different toolchains, architectures, C-libraries, thread-implementations and more. To help you with that, Buildroot provides the utils/test-pkg script.

Nothing describes the script better than its own help text [1]:

test-pkg: test-build a package against various toolchains and architectures

The supplied config snippet is appended to each toolchain config, the
resulting configuration is checked to ensure it still contains all options
specified in the snippet; if any is missing, the build is skipped, on the
assumption that the package under test requires a toolchain or architecture
feature that is missing.

In case failures are noticed, you can fix the package and just re-run the
same command again; it will re-run the test where it failed. If you did
specify a package (with -p), the package build dir will be removed first.

The list of toolchains is retrieved from support/config-fragments/autobuild/toolchain-configs.csv.
Only the external toolchains are tried, because building a Buildroot toolchain
would take too long. An alternative toolchains CSV file can be specified with
the -t option. This file should have lines consisting of the path to the
toolchain config fragment and the required host architecture, separated by a
comma. The config fragments should contain only the toolchain and architecture
settings.

By default, a useful subset of toolchains is tested. If needed, all
toolchains can be tested (-a), an arbitrary number of toolchains (-n
in order, -r for random).

Hands on

In these examples I'm going to build the CRIU package that i recently worked on.

First I will create a config snippet that contains the necessary options to enable my packet. In my case it's CRIU and HOST_PYTHON3:

     cat > criu.config <<EOF
     BR2_PACKAGE_HOST_PYTHON3=y
     BR2_PACKAGE_CRIU=y
     EOF

I can now start test-pkg and provide the config snippet:

     utils/test-pkg -c criu.config -p criu
                     bootlin-armv5-uclibc [1/6]: FAILED
                      bootlin-armv7-glibc [2/6]: FAILED
                    bootlin-armv7m-uclibc [3/6]: SKIPPED
                      bootlin-x86-64-musl [4/6]: OK
                       br-arm-full-static [5/6]: SKIPPED
                             sourcery-arm [6/6]: FAILED
     6 builds, 2 skipped, 3 build failed, 0 legal-info failed, 0
show-info failed

Some of them fails, the end of the build log is available at ~/br-test-pkg/*/logfile.

Now start read the logfiles and fix the errors for each failed test.

Once the test-pkg (without -a) will have no failures, it will be a good test to retry it with the -a (run all tests) option:

     utils/test-pkg -a -c criu.config -p criu

As soon as more and more tests passes, it takes unnecessary amount of time to run the same (successful) test again, so it's better to hand-pick those tests that actually fails.

To do this you may provide a specific list of toolchain configurations:

     cp support/config-fragments/autobuild/toolchain-configs.csv
criu-toolchains.csv
     # edit criu-toolchains.csv to keep your toolchains of interest.
     utils/test-pkg -a -t criu-toolchains.csv -c criu.config -p criu

This will retest only the toolchains kept in the csv.

Support for CRIU in Buildroot

Support for CRIU in Buildroot

A couple of months ago I started to evaluate [1] CRIU [2] for a project I'm working on. The project itself is using Buildroot to build and generate the root filesystem. Unfortunately, Buildroot lacks support for CRIU so there were some work to do.

/media/buildroot-plus-criu.png

To write the package was not straight forward. The package is only supported on certain architectures and the utils/test-pkg script failed for a few toolchains. Julien Olivain was really helpful to sort it out and he even wrote runtime scripts for it. Thanks for that.

I do not understand why projects still use custom Makefiles instead of CMake or Autotools though. Is is something essential that I've completely missed?

Kernel configuration

CRIU makes use of a lot of features that has to be enabled in the Linux kernel for full usage.

CONFIG_CHECKPOINT_RESTORE will be set by the package itself, but there are more configuration options that could be useful depending on how you intend to use the tool.

Relevant configuration options are:

General setup options
  • CONFIG_CHECKPOINT_RESTORE=y (Checkpoint/restore support)
  • CONFIG_NAMESPACES=y (Namespaces support)
  • CONFIG_UTS_NS=y (Namespaces support -> UTS namespace)
  • CONFIG_IPC_NS=y (Namespaces support -> IPC namespace)
  • CONFIG_SYSVIPC_SYSCTL=y
  • CONFIG_PID_NS=y (Namespaces support -> PID namespaces)
  • CONFIG_NET_NS=y (Namespaces support -> Network namespace)
  • CONFIG_FHANDLE=y (Open by fhandle syscalls)
  • CONFIG_EVENTFD=y (Enable eventfd() system call)
  • CONFIG_EPOLL=y (Enable eventpoll support)
  • CONFIG_RSEQ=y (Enable eventpoll support)
Networking support -> Networking options options for sock-diag subsystem
  • CONFIG_UNIX_DIAG=y (Unix domain sockets -> UNIX: socket monitoring interface)
  • CONFIG_INET_DIAG=y (TCP/IP networking -> INET: socket monitoring interface)
  • CONFIG_INET_UDP_DIAG=y (TCP/IP networking -> INET: socket monitoring interface -> UDP: socket monitoring interface)
  • CONFIG_PACKET_DIAG=y (Packet socket -> Packet: sockets monitoring interface)
  • CONFIG_NETLINK_DIAG=y (Netlink socket -> Netlink: sockets monitoring interface)
  • CONFIG_NETFILTER_XT_MARK=y (Networking support -> Networking options -> Network packet filtering framework (Netfilter) -> Core Netfilter Configuration -> Netfilter Xtables support (required for ip_tables) -> nfmark target and match support)
  • CONFIG_TUN=y (Networking support -> Universal TUN/TAP device driver support)

In the beginning of the project, CRIU had their own custom kernel which contained some experimental CRIU related patches. Nowadays many of those patches has been mainlined.

One such patch [3] that I missed in my current kernel verson (v5.10) was introduced in v5.12. It was related to how CRIU gets the process state and is essential to create a checkpoint of a running process:

commit 90f093fa8ea48e5d991332cee160b761423d55c1
Author: Piotr Figiel <figiel@google.com>
Date:   Fri Feb 26 14:51:56 2021 +0100

    rseq, ptrace: Add PTRACE_GET_RSEQ_CONFIGURATION request

    For userspace checkpoint and restore (C/R) a way of getting process state
    containing RSEQ configuration is needed.

    There are two ways this information is going to be used:
     - to re-enable RSEQ for threads which had it enabled before C/R
     - to detect if a thread was in a critical section during C/R

    Since C/R preserves TLS memory and addresses RSEQ ABI will be restored
    using the address registered before C/R.

    Detection whether the thread is in a critical section during C/R is needed
    to enforce behavior of RSEQ abort during C/R. Attaching with ptrace()
    before registers are dumped itself doesn't cause RSEQ abort.
    Restoring the instruction pointer within the critical section is
    problematic because rseq_cs may get cleared before the control is passed
    to the migrated application code leading to RSEQ invariants not being
    preserved. C/R code will use RSEQ ABI address to find the abort handler
    to which the instruction pointer needs to be set.

    To achieve above goals expose the RSEQ ABI address and the signature value
    with the new ptrace request PTRACE_GET_RSEQ_CONFIGURATION.

    This new ptrace request can also be used by debuggers so they're aware
    of stops within restartable sequences in progress.

    Signed-off-by: Piotr Figiel <figiel@google.com>
    Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
    Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
    Reviewed-by: Michal Miroslaw <emmir@google.com>
    Reviewed-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
    Acked-by: Oleg Nesterov <oleg@redhat.com>
    Link: https://lkml.kernel.org/r/20210226135156.1081606-1-figiel@google.com

With that said, to make use of CRIU with the latest features it's highly recommended to use a recent kernel.

And soon it will be available as a package in Buildroot.

TIL - Buildroot and LIBFOO_LINUX_CONFIG_FIXUPS

TIL - Buildroot and LIBFOO_LINUX_CONFIG_FIXUPS

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

Some applications in a Linux system depends on certain kernel features to work properly. I'm currently working on adding support for CRIU [1] in Buildroot [2] which has such requirements.

That's when I stumble upon the LIBFOO_LINUX_CONFIG_FIXUP variable.

LIBFOO_LINUX_CONFIG_FIXUPS

As the documentation [3] says:

* +LIBFOO_LINUX_CONFIG_FIXUPS+ lists the Linux kernel configuration
  options that are needed to build and use this package, and without
  which the package is fundamentally broken. This shall be a set of
  calls to one of the kconfig tweaking option: `KCONFIG_ENABLE_OPT`,
  `KCONFIG_DISABLE_OPT`, or `KCONFIG_SET_OPT`.
  This is seldom used, as package usually have no strict requirements on
  the kernel options.

This is a variable to tweak the kernel configuration from the package itself. In my case I had to:

define CRIU_LINUX_CONFIG_FIXUPS
         $(call KCONFIG_ENABLE_OPT,CONFIG_CHECKPOINT_RESTORE)
endef

To enable CONFIG_CHECKPOINT_RESTORE upon package selection.

TIL - Buildroot & BR_NO_CHECK_HASH_FOR

TIL - Buildroot & BR_NO_CHECK_HASH_FOR

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

In Buildroot [1], the integrity of (allmost) all downloaded packages is verified against a hash. Even packages that are fetched from a git repository is verified this way.

This is a good thing that no one really should work around.

Today I had a debug-session for one package which I locally cloned and frequently made changes to. I told Buildroot to fetch the source code for the package locally to speed up my iterations between new code changes and testing on target.

The hash did of course change after each code change, which became quite annoying.

BR_NO_CHECK_HASH_FOR

The BR_NO_CHECK_HASH_FOR was something I found by a coincidence when looking into the support/download/check-hash file.

No wonder why I have missed this one, it is not mention in the documentation at all:

[22:58:16]marcus@goliat:~/git/buildroot$ git grep BR_NO_CHECK_HASH_FOR docs/
[22:58:18]marcus@goliat:~/git/buildroot$

It does what it says - do not check hash for a certain list of files.

Here is an example on how the linux package make use of BR_NO_CHECK_HASH_FOR:

ifeq ($(BR2_LINUX_KERNEL)$(BR2_LINUX_KERNEL_LATEST_VERSION),y)
BR_NO_CHECK_HASH_FOR += $(LINUX_SOURCE)
endif

LINUX_PATCHES = $(call qstrip,$(BR2_LINUX_KERNEL_PATCH))

# We have no way to know the hashes for user-supplied patches.
BR_NO_CHECK_HASH_FOR += $(notdir $(LINUX_PATCHES))

Conclusion

I found this useful for my bug-hunting, but as a general rule, this is something that you probably should not use.

Buildroot: out-of-tree builds

Buildroot: out-of-tree builds

Like the Linux kernel, Buildroot supports building out-of-tree in a very similar way. This could be very useful when you have multiple configurations in the same Buildroot repository that you want to build without interference.

Consider the following example:

cd buildroot/
mkdir -p ../outputs/device{1,2}

make O=../outputs/device1 menuconfig
make O=../outputs/device1

make O=../outputs/device2 menuconfig
make O=../outputs/device2

Each output has its own .config so you may change the build configurations independently.

The big benefit compared with git worktree [1] or other multiple-instances-of-the-same-repository-approaches is that the dl directory is shared (without need to specify BR2_DL_DIR [2]) among all build directories.

The feature is documented in the Buildroot manual [3].

config utility for Buildroot

config utility for Buildroot

I'm using the ./scripts/config script in the Linux kernel tree a lot. The script is used to manipulate a .config file from the command line which is quite nice to be able to do.

I use it mostly to enable configurations from a script or as a part of automated tests.

Buildroot is also using KBuild as its configuration system so I adapted this script and submitted a patch. It's now part of the mainline.

commit f2c5c7a263d9d61faba80544e0f6ac6da03716ee
Author: Marcus Folkesson <marcus.folkesson@gmail.com>
Date:   Tue Sep 26 23:15:02 2017 +0200

utils/config: new script to manipulate .config files on the command line

Default prefix is set to `BR2_` but may be overridden by setting
BR2_PREFIX.

Example usage:

Enable `BR2_PACKAGE_GNUPG`:
./utils/config --package --enable GNUPG

Check state of config option `BR2_PACKAGE_GNUPG`:
./utils/config --package --state GNUPG
y

Enable `BR2_PACKAGE_GNUPG`:
./utils/config --package --disable GNUPG

Set `BR2_TARGET_GENERIC_ISSUE` to "Welcome":
./utils/config --set-str  TARGET_GENERIC_ISSUE "Welcome"

Copied from the Linux kernel (4.13-rc6) source code and adapted to
Buildroot.
Thanks to Andi Kleen who is the original author of this script.

Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>
[Arnout: merge the two tr invications]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

In addition to all commands found in he Linux kernel version, I also added the --package config as a shortcut to operate on packages.