Crosscompile libcamera for RPi

Goal

The goal is to cross-compile libcamera [1] and libcamera-apps [2] for Raspberry Pi using the latest Raspbian [3] (Bullseye) release. Usually you setup the root filesystem with Buildroot [4] or Yocto [5] and generate a SDK that you can use to compile your application. The Raspbian distribution does not come with a SDK so we have to setup our own.

We will use a Raspberry Pi 3b for this.

What is libcamera?

You can read about libcamera in a previous post [6].

Prepare the SD-card

We will boot the Raspberry Pi from a SD-card, so we first have to prepare it.

Download Raspbian Bullseye

You could either download the image [3] yourself or use rpi-imager. Only make sure that you chose the 64-bit version as that is what the toolchain we are going to use is built for.

To download and flash it yourself

wget https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-09-26/2022-09-22-raspios-bullseye-arm64-lite.img.xz
tar xf 2022-09-22-raspios-bullseye-arm64-lite.img.xz
sudo dd if=2022-09-22-raspios-bullseye-arm64-lite.img.xz of=/dev/mmcblk0 conv=sync

/dev/mmcblk0 will be overwritten, please double check that this is your SD-card dedicated for your Raspberry and not your wedding pictures.

I will stick to rpi-imager as it let you configure WLAN, enable SSH and set passwords in the configuration menu. I find it smooth.

/media/rpi-imager.jpg

As I'm using Arch Linux, rpi-imager is available as an AUR package

git clone https://aur.archlinux.org/rpi-imager.git
cd rpi-imager
makepkg -i

Other Debian-based distribution could just use apt-get

apt-get install rpi-imager

Enable UART

Once the image is written to the SD-card we can enable the UART. This is not necessary but I strongly prefer to have a serial port connected to the device I'm working with to see the boot log and get a getty.

Mount the boot partition and enable UART by write "enable_uart=1" to config.txt

sudo mount /dev/mmcblk0p1 mnt/disk/
echo enable_uart=1 | sudo tee -a mnt/disk/config.txt
sudo umount mnt/disk

Permit root login on SSH

Now it's time to power up the Raspberry Pi. Put the SD-card in the slot, power it up and login as the pi user, either via UART or SSH.

Permit root login in order to mount root filesystem via SSH

echo PermitRootLogin yes  | sudo tee -a /etc/ssh/sshd_config

Restart the SSH service

sudo systemctl restart sshd.service

Note that it's bad practice to let the root user login via SSH (especially with password). Either use SSH-keys or disable it later.

If you not allready have figured out which IP address the RPi has, grab it with ip addr

pi@raspberrypi:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:91:e6:2a brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.111/24 brd 192.168.1.255 scope global dynamic noprefixroute eth0
       valid_lft 409sec preferred_lft 334sec
    inet6 fe80::cfe6:1f35:c5b6:aa1a/64 scope link
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:c4:b3:7f brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.56/24 brd 192.168.1.255 scope global dynamic noprefixroute wlan0
       valid_lft 408sec preferred_lft 333sec
    inet6 fe80::61e6:aeb:2c0e:f31b/64 scope link
       valid_lft forever preferred_lft forever

My board has the IP 192.168.1.111 on the eth0 interface.

Set root password

The root user does not have a password by default, so set it

sudo passwd root

Install dependencies

Especially libcamera-apps have a lot of dependencies, install those using apt-get

sudo apt-get install libboost-program-options-dev libboost-dev libexif-dev libjpeg-dev libtiff-dev libpng-dev libdrm-dev libavcodec-dev libavdevice-dev

Prepare the host

Now we have everything on target in place so can switch back to the host system.

From now on, we will use environment variables to setup paths to all directories we will refer to. First, create those directories

mkdir rootfs staging tools

rootfs will be used for our network mounted root filesystem. staging will be our sysroot we compile against tools will contain our cross toolchain

Export the environment variables

export RPI_BASE=`pwd`
export RPI_ROOTFS=$RPI_BASE/rootfs
export RPI_STAGING=$RPI_BASE/staging
export RPI_TOOLS=$RPI_BASE/tools/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu/bin/

Install cross toolchain

Download cross toolchain with the same GCC version (10.2) as Raspian.

wget http://sources.buildroot.net/toolchain-external-arm-aarch64/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz
tar -xf gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz -C tools/

Add the toolchain to your $PATH

export PATH=$RPI_TOOLS:$PATH

Mount root filesystem

SSHFS (Secure SHell File System) [7] is a handy tool based in libfuse that let you mount a filesystem over SSH.

Mount the root filesystem

sshfs root@192.168.1.111:/ $RPI_ROOTFS

Prepare the staging directory

Here we will copy files from our root filesystem

mkdir -p $RPI_STAGING/usr/
cp -r $RPI_ROOTFS/usr/lib $RPI_STAGING/usr/
cp -r $RPI_ROOTFS/usr/include/ $RPI_STAGING/usr/

Now you may use ./rootfs as your sysroot. However, as the Rasbian image has C-library for both musl and glibc, the search pathes becomes hard to handle. We will look on how we could handle this later, but that is simply not worth it in my opinion. Instead, copy the glibc libraries to /usr/lib

cp $RPI_STAGING/usr/lib/aarch64-linux-gnu/* $RPI_STAGING/usr/lib/

We also need to create a symlink to /lib as the toolchain is looking for the linux loader in that directory

ln -s usr/lib/ $RPI_STAGING/lib

The the libpthread.so.6 is pointing to an aboslut path (will point to our host system). The result will be that the linker does not find the library and will fallback on the static linked library instead. That will not fall out good as the glibc is still dynamically linked... so create a new symlink

ln -sf libpthread.so.0 $RPI_STAGING/lib/libpthread.so

Cross compile libcamera

Cross compile libcamera is quite straight forward as it does use Meson as build system and do not have any external dependencies.

Clone the repository

git clone https://git.libcamera.org/libcamera/libcamera.git
cd libcamera

We need to create a toolchain file to instruct the Meson build system which toolchain it should use. Create aarch64.txt which contains

[binaries]
c = 'aarch64-none-linux-gnu-gcc'
cpp = 'aarch64-none-linux-gnu-g++'
ar = 'aarch64-none-linux-gnu-ar'
strip = 'aarch64-none-linux-gnu-strip'

[host_machine]
system = 'linux'
cpu_family = 'x86_64'
cpu = 'x86_64'
endian = 'little'

Now we could build

meson build -Dprefix=/usr/ --cross-file ./aarch64.txt
cd build
ninja

Install on both root filesystem and staging directory

DESTDIR=$RPI_ROOTFS ninja install
DESTDIR=$RPI_STAGING ninja install

Cross compile libcamera-apps

libcamera-apps on the other hand, have a lot of dependencies but as we have collected all of those into the staging directory, even this is pretty straight forward.

Just for your information, if we did not copy the glibc-libraries from /usr/lib/aarch-linux-gnu to /usr/lib, this would not be a straight trip at all.

Clone the repository

git clone https://github.com/raspberrypi/libcamera-apps.git
cd libcamera-apps

I'm not interresting in the preview application as it needs Qt5, so I will remove it

sed -i "/add_subdirectory(preview)/d" CMakeLists.txt

I also did remove libcamera-raw and libcamera-vid from apps/CMakeLists.txt as those have dependencies that I do not want on my target.

libcamera-apps does use CMake as build system, and we need to create a toolchain file for this as well. Create the file aarch64.cmake which contains

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR arm)

set(CMAKE_SYSROOT $ENV{RPI_STAGING})

#We need to point to the glibc-headers
set(CMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES $ENV{RPI_STAGING}/usr/include/aarch64-linux-gnu/)

set(CMAKE_C_COMPILER $ENV{RPI_TOOLS}/aarch64-none-linux-gnu-gcc)
set(CMAKE_CXX_COMPILER $ENV{RPI_TOOLS}/aarch64-none-linux-gnu-g++)

#Let pkg-config look in our root
SET(ENV{PKG_CONFIG_LIBDIR} ${CMAKE_FIND_ROOT_PATH}/lib/pkgconfig/)

set(CMAKE_FIND_ROOT_PATH "${CMAKE_SYSROOT}")
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

Now we are ready to build

mkdir build
cd build
cmake -DCMAKE_TOOLCHAIN_FILE=../aarch64.cmake ../
make

And install

make DESTDIR=$RPI_ROOTFS install

We do only need to install to our root filessytem as it should run on target.

Test on target

We can now run the applications on target and verify that it's our newly built app:

pi@raspberrypi:/usr/local/bin$ LD_LIBRARY_PATH=/usr/local/lib ./libcamera-jpeg  --version
libcamera-apps build: 80f17befef34-intree-dirty 26-11-2022 (15:55:47)
libcamera build: v0.0.2+34-b35f04b3

For the good sake, we could also take an image

pi@raspberrypi:/usr/local/bin$ LD_LIBRARY_PATH=/usr/local/lib ./libcamera-jpeg  -o /tmp/test1.jpg
/media/libcamera-rpi-small.jpg

Now you can disable root login for SSH

sed -i "/PermitRootLogin/d" /etc/ssh/sshd_config
sudo systemctl restart sshd.service

What if...

...we should not copy /usr/lib/aarch64-linux-gnu to /usr/lib but keep it as it's? It would be nice to get rid of the staging directory and only use the root filesystem for cross compiling.

That was my first intention, but it turned out to be really troublesome, even if I got it to work at last.

In short what is needed to be done is:

  • Export PKG_CONFIG_SYSROOT_DIR to $RPI_ROOTFS
  • Export PKG_CONFDIG_PATH to $RPI_ROOTFS/usr/lib/aarch64-linux-gnu/pkgconfig to make it find pkgconfig files
  • Export BOOST_ROOT to $RPI_ROOTFS to give FindBoost.cmake a hint of where to look
  • Export BOOST_LIBRARYDIR to $RPI_ROOTFS/usr/lib/aarch64-linux-gnu
  • Set CMAKE_LIBRARY_PATH to look into $RPI_ROOTFS/usr/lib/aarch64_linux-gnu
  • Set CMAKE_CXX_FLAGS_INIT, CMAKE_C_FLAGS_INIT and CMAKE_EXE_LINKER_FLAGS to search $RPI_ROOTFS/usr/lib/aarch64-linux-gnu for libraries

There are simply too much special and I think it's not worth it.

Conclusion

This sounds like a simple thing, but it actually took quite a while to get it working. Mostly because of the that the Raspian distribution supports both and glibc/musl so that libraries, pkgconfig files and headers end up in non-standard search paths.

I find it quite strange that there are no SDK available for the Raspian images, it would help development against the platform a lot.