Crosscompile libcamera for RPi

Crosscompile libcamera for RPi

Goal

The goal is to cross-compile libcamera [1] and libcamera-apps [2] for Raspberry Pi using the latest Raspbian [3] (Bullseye) release. Usually you setup the root filesystem with Buildroot [4] or Yocto [5] and generate a SDK that you can use to compile your application. The Raspbian distribution does not come with a SDK so we have to setup our own.

We will use a Raspberry Pi 3b for this.

What is libcamera?

You can read about libcamera in a previous post [6].

Prepare the SD-card

We will boot the Raspberry Pi from a SD-card, so we first have to prepare it.

Download Raspbian Bullseye

You could either download the image [3] yourself or use rpi-imager. Only make sure that you chose the 64-bit version as that is what the toolchain we are going to use is built for.

To download and flash it yourself

wget https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-09-26/2022-09-22-raspios-bullseye-arm64-lite.img.xz
tar xf 2022-09-22-raspios-bullseye-arm64-lite.img.xz
sudo dd if=2022-09-22-raspios-bullseye-arm64-lite.img.xz of=/dev/mmcblk0 conv=sync

/dev/mmcblk0 will be overwritten, please double check that this is your SD-card dedicated for your Raspberry and not your wedding pictures.

I will stick to rpi-imager as it let you configure WLAN, enable SSH and set passwords in the configuration menu. I find it smooth.

/media/rpi-imager.jpg

As I'm using Arch Linux, rpi-imager is available as an AUR package

git clone https://aur.archlinux.org/rpi-imager.git
cd rpi-imager
makepkg -i

Other Debian-based distribution could just use apt-get

apt-get install rpi-imager

Enable UART

Once the image is written to the SD-card we can enable the UART. This is not necessary but I strongly prefer to have a serial port connected to the device I'm working with to see the boot log and get a getty.

Mount the boot partition and enable UART by write "enable_uart=1" to config.txt

sudo mount /dev/mmcblk0p1 mnt/disk/
echo enable_uart=1 | sudo tee -a mnt/disk/config.txt
sudo umount mnt/disk

Permit root login on SSH

Now it's time to power up the Raspberry Pi. Put the SD-card in the slot, power it up and login as the pi user, either via UART or SSH.

Permit root login in order to mount root filesystem via SSH

echo PermitRootLogin yes  | sudo tee -a /etc/ssh/sshd_config

Restart the SSH service

sudo systemctl restart sshd.service

Note that it's bad practice to let the root user login via SSH (especially with password). Either use SSH-keys or disable it later.

If you not allready have figured out which IP address the RPi has, grab it with ip addr

pi@raspberrypi:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:91:e6:2a brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.111/24 brd 192.168.1.255 scope global dynamic noprefixroute eth0
       valid_lft 409sec preferred_lft 334sec
    inet6 fe80::cfe6:1f35:c5b6:aa1a/64 scope link
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:c4:b3:7f brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.56/24 brd 192.168.1.255 scope global dynamic noprefixroute wlan0
       valid_lft 408sec preferred_lft 333sec
    inet6 fe80::61e6:aeb:2c0e:f31b/64 scope link
       valid_lft forever preferred_lft forever

My board has the IP 192.168.1.111 on the eth0 interface.

Set root password

The root user does not have a password by default, so set it

sudo passwd root

Install dependencies

Especially libcamera-apps have a lot of dependencies, install those using apt-get

sudo apt-get install libboost-program-options-dev libboost-dev libexif-dev libjpeg-dev libtiff-dev libpng-dev libdrm-dev libavcodec-dev libavdevice-dev

Prepare the host

Now we have everything on target in place so can switch back to the host system.

From now on, we will use environment variables to setup paths to all directories we will refer to. First, create those directories

mkdir rootfs staging tools

rootfs will be used for our network mounted root filesystem. staging will be our sysroot we compile against tools will contain our cross toolchain

Export the environment variables

export RPI_BASE=`pwd`
export RPI_ROOTFS=$RPI_BASE/rootfs
export RPI_STAGING=$RPI_BASE/staging
export RPI_TOOLS=$RPI_BASE/tools/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu/bin/

Install cross toolchain

Download cross toolchain with the same GCC version (10.2) as Raspian.

wget http://sources.buildroot.net/toolchain-external-arm-aarch64/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz
tar -xf gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz -C tools/

Add the toolchain to your $PATH

export PATH=$RPI_TOOLS:$PATH

Mount root filesystem

SSHFS (Secure SHell File System) [7] is a handy tool based in libfuse that let you mount a filesystem over SSH.

Mount the root filesystem

sshfs root@192.168.1.111:/ $RPI_ROOTFS

Prepare the staging directory

Here we will copy files from our root filesystem

mkdir -p $RPI_STAGING/usr/
cp -r $RPI_ROOTFS/usr/lib $RPI_STAGING/usr/
cp -r $RPI_ROOTFS/usr/include/ $RPI_STAGING/usr/

Now you may use ./rootfs as your sysroot. However, as the Rasbian image has C-library for both musl and glibc, the search pathes becomes hard to handle. We will look on how we could handle this later, but that is simply not worth it in my opinion. Instead, copy the glibc libraries to /usr/lib

cp $RPI_STAGING/usr/lib/aarch64-linux-gnu/* $RPI_STAGING/usr/lib/

We also need to create a symlink to /lib as the toolchain is looking for the linux loader in that directory

ln -s usr/lib/ $RPI_STAGING/lib

The the libpthread.so.6 is pointing to an aboslut path (will point to our host system). The result will be that the linker does not find the library and will fallback on the static linked library instead. That will not fall out good as the glibc is still dynamically linked... so create a new symlink

ln -sf libpthread.so.0 $RPI_STAGING/lib/libpthread.so

Cross compile libcamera

Cross compile libcamera is quite straight forward as it does use Meson as build system and do not have any external dependencies.

Clone the repository

git clone https://git.libcamera.org/libcamera/libcamera.git
cd libcamera

We need to create a toolchain file to instruct the Meson build system which toolchain it should use. Create aarch64.txt which contains

[binaries]
c = 'aarch64-none-linux-gnu-gcc'
cpp = 'aarch64-none-linux-gnu-g++'
ar = 'aarch64-none-linux-gnu-ar'
strip = 'aarch64-none-linux-gnu-strip'

[host_machine]
system = 'linux'
cpu_family = 'x86_64'
cpu = 'x86_64'
endian = 'little'

Now we could build

meson build -Dprefix=/usr/ --cross-file ./aarch64.txt
cd build
ninja

Install on both root filesystem and staging directory

DESTDIR=$RPI_ROOTFS ninja install
DESTDIR=$RPI_STAGING ninja install

Cross compile libcamera-apps

libcamera-apps on the other hand, have a lot of dependencies but as we have collected all of those into the staging directory, even this is pretty straight forward.

Just for your information, if we did not copy the glibc-libraries from /usr/lib/aarch-linux-gnu to /usr/lib, this would not be a straight trip at all.

Clone the repository

git clone https://github.com/raspberrypi/libcamera-apps.git
cd libcamera-apps

I'm not interresting in the preview application as it needs Qt5, so I will remove it

sed -i "/add_subdirectory(preview)/d" CMakeLists.txt

I also did remove libcamera-raw and libcamera-vid from apps/CMakeLists.txt as those have dependencies that I do not want on my target.

libcamera-apps does use CMake as build system, and we need to create a toolchain file for this as well. Create the file aarch64.cmake which contains

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR arm)

set(CMAKE_SYSROOT $ENV{RPI_STAGING})

#We need to point to the glibc-headers
set(CMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES $ENV{RPI_STAGING}/usr/include/aarch64-linux-gnu/)

set(CMAKE_C_COMPILER $ENV{RPI_TOOLS}/aarch64-none-linux-gnu-gcc)
set(CMAKE_CXX_COMPILER $ENV{RPI_TOOLS}/aarch64-none-linux-gnu-g++)

#Let pkg-config look in our root
SET(ENV{PKG_CONFIG_LIBDIR} ${CMAKE_FIND_ROOT_PATH}/lib/pkgconfig/)

set(CMAKE_FIND_ROOT_PATH "${CMAKE_SYSROOT}")
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

Now we are ready to build

mkdir build
cd build
cmake -DCMAKE_TOOLCHAIN_FILE=../aarch64.cmake ../
make

And install

make DESTDIR=$RPI_ROOTFS install

We do only need to install to our root filessytem as it should run on target.

Test on target

We can now run the applications on target and verify that it's our newly built app:

pi@raspberrypi:/usr/local/bin$ LD_LIBRARY_PATH=/usr/local/lib ./libcamera-jpeg  --version
libcamera-apps build: 80f17befef34-intree-dirty 26-11-2022 (15:55:47)
libcamera build: v0.0.2+34-b35f04b3

For the good sake, we could also take an image

pi@raspberrypi:/usr/local/bin$ LD_LIBRARY_PATH=/usr/local/lib ./libcamera-jpeg  -o /tmp/test1.jpg
/media/libcamera-rpi-small.jpg

Now you can disable root login for SSH

sed -i "/PermitRootLogin/d" /etc/ssh/sshd_config
sudo systemctl restart sshd.service

What if...

...we should not copy /usr/lib/aarch64-linux-gnu to /usr/lib but keep it as it's? It would be nice to get rid of the staging directory and only use the root filesystem for cross compiling.

That was my first intention, but it turned out to be really troublesome, even if I got it to work at last.

In short what is needed to be done is:

  • Export PKG_CONFIG_SYSROOT_DIR to $RPI_ROOTFS
  • Export PKG_CONFDIG_PATH to $RPI_ROOTFS/usr/lib/aarch64-linux-gnu/pkgconfig to make it find pkgconfig files
  • Export BOOST_ROOT to $RPI_ROOTFS to give FindBoost.cmake a hint of where to look
  • Export BOOST_LIBRARYDIR to $RPI_ROOTFS/usr/lib/aarch64-linux-gnu
  • Set CMAKE_LIBRARY_PATH to look into $RPI_ROOTFS/usr/lib/aarch64_linux-gnu
  • Set CMAKE_CXX_FLAGS_INIT, CMAKE_C_FLAGS_INIT and CMAKE_EXE_LINKER_FLAGS to search $RPI_ROOTFS/usr/lib/aarch64-linux-gnu for libraries

There are simply too much special and I think it's not worth it.

Conclusion

This sounds like a simple thing, but it actually took quite a while to get it working. Mostly because of the that the Raspian distribution supports both and glibc/musl so that libraries, pkgconfig files and headers end up in non-standard search paths.

I find it quite strange that there are no SDK available for the Raspian images, it would help development against the platform a lot.

What is libcamera and why should you use it?

What is libcamera and why should you use it

Read out a picture from camera

Once in a time, video devices was not that complex. To use a camera back then, your application software could iterated through /dev/video* devices and pick the camera that you want and then immediately start using it. You could query which pixel formats, frame rates, resolutions and all other properties that are supported by the camera. You could even easily change it if you want.

This still works for some cameras, basically every USB camera and most laptop cameras still works that way.

The problem, especially in embedded systems, is that there is no such thing as "the camera" anymore. The camera system is rather a complex pipeline of different image processing nodes that the image data traverse through to be shaped as you want. Even if the result of this pipeline will end up in a video device, you cannot configure things like cropping, resolution etc. directly on that device as you used to. Instead, you have to use the media controller API to configure and link each of these nodes to build up your pipeline.

To show how it may look like; this is a graph that I had in a previous post [3]:

/media/media-ctl-graph.png

What is libcamera?

/media/libcamera-banner.png

This is how libcamera is described on their website [1]

libcamera is an open source camera stack for many platforms with a core userspace library, and support from the Linux kernel APIs and drivers already in place.
It aims to control the complexity of embedded camera hardware by providing an intuitive API and method of separating untrusted vendor code from the open source core.

libcamera aims to encourage the development of new embedded camera applications by limiting the complexity that developers have to deal with.
The interface is designed around the way that modern embedded camera hardware works.

First time I heard about libcamera was on the Embedded Linux Conference 2019 where Jacopo Mondi had a talk [2] about the Public API for the first stable libcamera release. I have been working with cameras in several embedded Linux products and know for sure how complex [3] these little beast could be. The configuration also differ depending on which platform or camera you are using as there is no common way to setup the image pipe. You will soon have special cases for all your platform variants in your application. Which is not what we strive for.

libcamera is trying to solve this by provide one library that takes care of all that complexity for you.

For example, if you want to adjust a simple thing, say contrast, of a IMX219 camera module connected to a Raspberry Pi. To do that without libcamera, you first have to setup a proper image pipeline that takes the camera module, connect it to the several ISP (Image Signal Processing) blocks that your processor offers in order to get the right image format, resolution and so on. Somewhere between all these configuring, you realise that the camera module nor the ISPs have support for adjust the contrast. Too bad. To achieve this you have to take the image, pass it to a self-written contrast algorithm, create a gamma curve that the IPA (Image Processing Algorithm) understands and actually set gamma. Yes, the contrast is adjusted with a gamma curve for that particular camera on Raspberry Pi. ( Have a look at the implementation of that IPA block [7] for Raspberry Pi )

This is exactly the stuff libcamera understands and abstract for the user. libcamera will figure out what graph it has to build depending on what you want do to and which processing operations that are available at your various nodes. The application that is using libcamera for the video device will set contrast for all cameras and platforms. After all, that is what you wanted.

Camera Stack

As the libcamera library is fully implemented in userspace and use already existing kernel interfaces for communication with hardware, you will need no extra underlying support in terms of separate drivers or kernel support.

libcamera itself exposes several API's depending on how the application want to interface the camera. It even have a V4L2 compatiblity layer to emulate a high-level V4L2 camera device to make a smooth transition for all those V4L2 applications out there.

/media/libcamera-layer.png

Read more about the camera stack in the libcamera documentation [4].

Conclusion

I really like this project and I think we need an open-source stack that supports many platforms. This vendor-specific drivers/libraries/IPAs-situation we are in right now is not sustainable at all. It takes too much effort to evaluate a few cameras of different vendors just because all vendors has their own way to control the camera with their own closed-source and platform specific layers. Been there done that.

For those vendors that do not want to open-source their secret image processing algorithms, libcamera uses a plugin system for IPA modules which let vendors keep their secrets but still be compatible with libcamera. All open-source modules are identified based on digital signatures, while closed-source modules are instead isolated inside a sandbox environment with restricted access to the system. A Win-Win concept.

The project itself is still quite young and need more work to support more platforms and cameras, but the ground is stable. Raspberry Pi is now a common used platform, both in commercial and hobby, and the fact that Raspberry Pi Foundation has chosen libcamera as their primary camera system [8] must tell us something.