Use custom EDID in Linux

Use custom EDID in Linux

Extended Display Identification Data (EDID) is a metadata format for display devices to describe their capabilities such as resolution, display size, timing, bit depth and update frequency. It is a 128-byte (EDID) or 256-byte (Enhanced-EDID) structure transferred from the display device over the Display Data Channel (DDC) protocol, which is a layer on top of the I2C specification.

The EDID is accessible via the I2C address 0x50 and can usually be read even if the display is turned off, which is quite nice.

Before Video Electronics Standard Association (VESA) came up with this standard, there were multiple non-standard ways out there to provide some kind of basic identification for video device.

Handle all these non-standard ways is of course an unmanageable situation. In that good old days we had to explicitly set all graphics parameters in the xorg.conf file.

Hooray for standards!

Read out the EDID structure

The EDID structure is available for DRM (Direct Rendring Manager) devices via sysfs in raw binary format:

$od  -Anone -t x1 /sys/devices/pci0000:00/0000:00:02.0/drm/card1/card1-DP-4/edid
     00 ff ff ff ff ff ff 00 41 0c c9 c0 ae 00 00 00
     1a 17 01 03 80 3c 22 78 2a 25 95 a9 54 4f a1 26
     0a 50 54 bd 4b 00 d1 00 d1 c0 81 80 95 0f 95 00
     b3 00 81 c0 a9 40 56 5e 00 a0 a0 a0 29 50 30 20
     35 00 55 50 21 00 00 1e 00 00 00 ff 00 41 55 34
     31 33 32 36 30 30 30 31 37 34 00 00 00 fc 00 50
     68 69 6c 69 70 73 20 32 37 32 43 34 00 00 00 fd
     00 32 4c 1e 63 21 00 0a 20 20 20 20 20 20 00 ac

read_edid [1] provide some tools to retrieve and interpret monitor specifications using the VESA DDC protocol. parse-edid is part of this package and we can use it to parse the EDID structure above:

$ parse-edid < /sys/devices/pci0000:00/0000:00:02.0/drm/card1/card1-DP-4/edid
Checksum Correct

Section "Monitor"
    Identifier "Philips 272C4"
    ModelName "Philips 272C4"
    VendorName "PHL"
    # Monitor Manufactured week 26 of 2013
    # EDID version 1.3
    # Digital Display
    DisplaySize 600 340
    Gamma 2.20
    Option "DPMS" "true"
    Horizsync 30-99
    VertRefresh 50-76
    # Maximum pixel clock is 330MHz
    #Not giving standard mode: 1920x1200, 60Hz
    #Not giving standard mode: 1920x1080, 60Hz
    #Not giving standard mode: 1280x1024, 60Hz
    #Not giving standard mode: 1440x900, 75Hz
    #Not giving standard mode: 1440x900, 60Hz
    #Not giving standard mode: 1680x1050, 60Hz
    #Not giving standard mode: 1280x720, 60Hz
    #Not giving standard mode: 1600x1200, 60Hz
    Modeline        "Mode 0" +hsync +vsync
EndSection

This is the EDID for my Philips Monitor.

Provide custom EDID structure to DRM

I'm working with a custom projector (yes, projectors are display devices too) board for an embedded Linux system. Unfortunately, the processor has an errata for the DDC channel which causes the retrieved EDID structure to be corrupt, so I have to manually provide EDID information for the DRM layer.

For such situations, the kernel has introduced the CONFIG_DRM_LOAD_EDID_FIRMWARE configuration item. It let you provide a individually prepared EDID data in the lib/firmware directory to be loaded instead of retrieving it on the DDC channel. The functionality is disabled by default as it is mostly a workaround for broken hardware, and you will, luckily enough, have to search hard to find such a hardware these days.

The sources also contains a few built-in [2] structures for commonly used screen resolutions for us to use:

#define GENERIC_EDIDS 6
static const char * const generic_edid_name[GENERIC_EDIDS] = {
    "edid/800x600.bin",
    "edid/1024x768.bin",
    "edid/1280x1024.bin",
    "edid/1600x1200.bin",
    "edid/1680x1050.bin",
    "edid/1920x1080.bin",
};

See the kernel documentation [3] for more details.

Use the custom EDID structure

We could either place our custom EDID data in /lib/firmware/edid/ or use one of those build-in structures. Either way, pass drm_kms_helper.edid_firmware pointing to the right structure as argument to the kernel.

Example on bootargs that use the built-in 800x600 EDID structure:

drm_kms_helper.edid_firmware=edid/800x600.bin

Here is my projector in action showing the Qt Analog Clock [4] example.

/media/edid-clock.jpg

(Yes, crappy image, it looks much better IRL)

Audio and Embedded Linux

Audio and Embedded Linux

Brief

Last time I used wrote kernel drivers for the ASoC (ALSA System on Chip) subsystem, the functionality was split up into these parts:

  • Platform class driver that defines the SoC audio interface for the actual CPU itself. This includes both the DAI (Digital Audio Interface) and any potential audio muxes (e.g. i.MX6 has its AUDMUX).
  • CODEC class driver that controls the actual CODEC.
  • Machine drivers that is the magic glue between the SoC and the CODEC which connect the both interfaces. Such driver had to be written for each SoC-CODEC combination, and that does not scale very well.

Nowadays, most of the CODEC class drivers is now adapted to be described with the simple-audio-card [1] in a device tree, which will completely replace the machine drivers.

The goal with this post is to describe my work to setup a a 20W mono class-D audio amplifier to work with a i.MX8MM board.

General

The configuration of the CODEC is usually done on a I2C bus, even if other simple busses like SPI could be used as well. When configuration is sent over this simple bus, the audio data is sent over a complete different bus.

Audio data could be transferred in many different formats such as AC97, PCM or I2S.

To abstract this bus and handle it in a common way, we will just call it DAI, for Digital Audio Interface.

Different SoCs have of course different names on this as well. For example, Texas Instruments has its McASP, NXP uses SSI, Atmel SSC and so on.. We call it DAI all over.

Serial audio formats

AC97

AC97 is a commonly found interface on many PC cards, it is not that popular in embedded devices though. It is a five wire interface with:

  • A reset line
  • DATA_OUT for playback
  • SDATA_IN for capture
  • BCLK as bit clock, which is always driven by the CODEC

See the specification [4] for further reading.

I2S

I2S is a comman 5 wire DAI often used in embedded systems. The Tx (SDOUT) and Rx (SDIN) lines are used for audio transmission while the bit and frame clock are used for synchronization.

The signals are:

  • Master clock or system clock, often referred to as MCLK, is the clock which the other clocks is derived from. This also clock the CODEC.
  • Bit clock, often referred to as BCK or BCLK, varies depending on the sample rate.
  • Frame clock, often referred to ass LRCLK (Left-Right Clock), FCLK (Frame clock) or WCLK (Word clock).
  • Audio out, SDOUT
  • Audio In, SDIN

The relationchip between BCLK and LRCLK is

bclk = (sample rate) * Nchannels * (bit depth)

Some CODECs are able to use BCLK as their only clock, which leaving MCLK as optional. The CODEC we will use does supports this and is something we have to use due to HW constraints in number of available signals that should fit in a connector.

This is an illustration of the timing on the I2S bus with 64 BCLKs per LRCLK. Borrowed from the datasheet [5]:

/media/i2s.jpg

I2S could be used with TDM format timing to support more audio channels on the same I2S bus. The timing will then look like this [5] :

/media/i2s-tdm.jpg

PCM

PCM is a 4 wire interface that is quite similiar to I2S. Same same but different.

Clocks

We have several clocks such as bit clock, frame clock and master clock. It is not written in stone which endpoint of the DAI that should generate these clocks, it is up to us to decide.

Either the SoC or CODEC generates some or all of the clocks, called clock master (e.g. bit clock master or frame clock master).

It is often easiest to let the codec generate all clocks, but some SoCs has specialized audio PLLs for this. In our case, the SoC will be clock master.

The Hardware

The SoC

The board we are going to use is an evaluation board for a i.MX8MM module [2]. The CPU module supports two I2S busses and we are going to use one of them.

/media/sm2simx8m.jpg

The CODEC

The CODEC we will use is is the TAS5720L [3] from Texas Instruments which has been supported in mainline since v4.6.

/media/tas5720l.jpg

The TAS5720L device Serial Audio Interface (SAIF) supports a variety of audio formats including I2S, left-justified and Right justified. It also supports the time division multiplexed (TDM) format that is capable of transporting up to 8 channels of audio data on a single bus.

It uses I2C as configuration interface.

We will use I2S with TDM as DAI and I2C as configuration interface.

The Software

As we mostly got rid of the machine drivers and can describe the CODEC bindings using device tree, the setup is mostly a exercise in device tree writing rather than C.

The device tree node to setup the sound card is simple-audio-card [6].

SAI node

The Synchronous Audio Interface (SAI) module is the HW part of the i.MX8 SoC that are used to generate the digital audio.

We are going to use the SAI5 interface as it is routed from the sm2s-imx8mm module. The node is properly configured in an interface (.dtsi)-file, so we only have to enable it:

&sai5 {
    status = "okay";
};

CODEC node

The TAS5720L is conneced to the I2C3 bus and respond to the slave address 0x6c. Besides the compatible and reg properties, the node also requires phandle for a 3V3 suppy that supplies the digital circuitry and a phandly for the Class-D amp and analog part.

The hardware does not have such controllable supplies so we have to create fixed regulators for that:

/ {
    reg_audio_p: regulator-audio {
        compatible = "regulator-fixed";
        regulator-name = "audio power";
        pinctrl-names = "default";
        regulator-min-microvolt = <12000000>;
        regulator-max-microvolt = <12000000>;
    };

    reg_audio_d: regulator-audio {
        compatible = "regulator-fixed";
        regulator-name = "audio digital";
        pinctrl-names = "default";
        regulator-min-microvolt = <3300000>;
        regulator-max-microvolt = <3300000>;
    };
};

And thedevice node for the CODEC itself:

&i2c3 {

    tas5720: tas5720@6c {
            #sound-dai-cells = <0>;
            reg = <0x6c>;
            compatible = "ti,tas5720";

            dvdd-supply = <&reg_audio_d>;
            pvdd-supply = <&reg_audio_p>;
    };
};

Sound node

Now it is time to setup the sound node!

First we have to specify wich audio format we intend to use by setting simple-audio-card,format to i2s.

We also have to setup the two DAIs (CPU & CODEC) that we are going to use.

This is done by creating subnodes and refer to the SAI module node and CODEC node as sound-dai respectively.

These subnodes are referred to when assign frame-master and bitclock-master in the sound node. As we want the SoC to generate both frame- and bit-clock, set cpudai as clock master for both.

/ {
    sound-tas5720 {
        compatible = "simple-audio-card";
        simple-audio-card,name = "tas5720-audio";
        simple-audio-card,format = "i2s";
        simple-audio-card,frame-master = <&cpudai>;
        simple-audio-card,bitclock-master = <&cpudai>;

        cpudai: simple-audio-card,cpu {
            sound-dai = <&sai5>;
            clocks = <&clk IMX8MM_CLK_SAI5_ROOT>;

        };

        simple-audio-card,codec {
            sound-dai = <&tas5720>;
            clocks = <&clk IMX8MM_CLK_SAI5_ROOT>;
        };
    };
};

Sound test

Now we should have everything in place!

Lets use speaker-test, which is part of alsa-utils [8] to test our setup.

root@imx8board:~# speaker-test

speaker-test 1.2.5.1

Playback device is default
Stream parameters are 44000Hz, S16_LE, 1 channels
Using 16 octav es of pink noise
[   12.257438] fsl-sai 30050000.sai: failed to derive required Tx rate: 1411200

That did not turn out well.

Debug clock signals

Lets look what our clock tree looks like:

root@imx8board:~# cat /sys/kernel/debug/clk/clk_summary
    ...
    audio_pll2_ref_sel                0        0        0    24000000          0     0  50000
       audio_pll2                     0        0        0   361267200          0     0  50000
          audio_pll2_bypass           0        0        0   361267200          0     0  50000
             audio_pll2_out           0        0        0   361267200          0     0  50000
    audio_pll1_ref_sel                0        0        0    24000000          0     0  50000
       audio_pll1                     0        0        0   393216000          0     0  50000
          audio_pll1_bypass           0        0        0   393216000          0     0  50000
             audio_pll1_out           0        0        0   393216000          0     0  50000
                sai5                  0        0        0    24576000          0     0  50000
                   sai5_root_clk       0        0        0    24576000          0     0  50000
    ...

The sai5 clock is running at 24576000Hz, and indeed, it is hard to find a working clock divider to get 1411200Hz.

audio_pll2 @ 361267200 looks better. 361267200/1411200=256, allmost perfect!

Then we need to reparent the sai5 module, this is done in the device tree as well:

&sai5 {
    status = "okay";
    assigned-clock-parents = <&clk IMX8MM_AUDIO_PLL2_OUT>;
    assigned-clock-rates = <11289600>;
};

Here is our new clock tree:

root@imx8board:~# cat /sys/kernel/debug/clk/clk_summary
    ...
    audio_pll2_ref_sel                0        0        0    24000000          0     0  50000
       audio_pll2                     0        0        0   361267200          0     0  50000
          audio_pll2_bypass           0        0        0   361267200          0     0  50000
             audio_pll2_out           0        0        0   361267200          0     0  50000
                sai5                  0        0        0    11289600          0     0  50000
                   sai5_root_clk       0        0        0    11289600          0     0  50000
    ...

We can see that the frequency is right and also that we now derive our clock from audio_pll2_out instead of audio_pll1.

The speaker-test software is also happier:

root@imx8board:~# speaker-test

speaker-test 1.2.5.1

Playback device is default
Stream parameters are 44000Hz, S16_LE, 1 channels
Using 16 octaves of pink noise
Rate set to 44000Hz (requested 44000Hz)
Buffer size range from 3840 to 5760
Period size range from 1920 to 1920
Using max buffer size 5760
Periods = 4
was set period_size = 1920
was set buffer_size = 5760
 0 - Front Left

Great!

Use BCLK as MCLK

Due to my hardware constraints, I need to use the bit clock as master clock. If we look in the datasheet [5] :

/media/tas5720-1.png

If the BCLK to LRCLK ratio is 64, we could tie MCLK directly to our BCLK!

We already know our BCLK, it is 1411200Hz, and the frame clock (LRCLK) is the same as the sample rate (44kHz). We could verify that with the oscilloscope.

Bitclock:

/media/bitclock1.png

Frameclock:

/media/frameclock.png

That is not a ratio of 64.

There is not much to do about the frame clock, it will stick to the sample rate. If we make use of TDM though, we can make the bit clock running faster with the same frame clock!

Lets add 2 TDM slots @ 32bit width:

/ {
    sound-tas5720 {
        compatible = "simple-audio-card";
        simple-audio-card,name = "tas5720-audio";
        simple-audio-card,format = "i2s";
        simple-audio-card,frame-master = <&cpudai>;
        simple-audio-card,bitclock-master = <&cpudai>;

        cpudai: simple-audio-card,cpu {
            sound-dai = <&sai5>;
            clocks = <&clk IMX8MM_CLK_SAI5_ROOT>;
            dai-tdm-slot-num = <2>;
            dai-tdm-slot-width = <32>;
        };

        simple-audio-card,codec {
            sound-dai = <&tas5720>;
            clocks = <&clk IMX8MM_CLK_SAI5_ROOT>;
        };
    };
};

Verify the bitclock:

/media/bitclock1.png

Lets calculate: 2820000/44000 ~= 64! We have reached our goal!

Final device tree setup

This is what the final device tree looks like:

/ {
    sound-tas5720 {
        compatible = "simple-audio-card";
        simple-audio-card,name = "tas5720-audio";
        simple-audio-card,format = "i2s";
        simple-audio-card,frame-master = <&cpudai>;
        simple-audio-card,bitclock-master = <&cpudai>;

        cpudai: simple-audio-card,cpu {
            sound-dai = <&sai5>;
            clocks = <&clk IMX8MM_CLK_SAI5_ROOT>;
            dai-tdm-slot-num = <2>;
            dai-tdm-slot-width = <32>;
        };

        simple-audio-card,codec {
            sound-dai = <&tas5720>;
            clocks = <&clk IMX8MM_CLK_SAI5_ROOT>;
        };
    };

    reg_audio_p: regulator-audio {
        compatible = "regulator-fixed";
        regulator-name = "audio power";
        pinctrl-names = "default";
        regulator-min-microvolt = <12000000>;
        regulator-max-microvolt = <12000000>;
    };

    reg_audio_d: regulator-audio {
        compatible = "regulator-fixed";
        regulator-name = "audio digital";
        pinctrl-names = "default";
        regulator-min-microvolt = <3300000>;
        regulator-max-microvolt = <3300000>;
    };

};

&i2c3 {

    tas5720: tas5720@6c {
            #sound-dai-cells = <0>;
            reg = <0x6c>;
            compatible = "ti,tas5720";

            dvdd-supply = <&reg_audio_d>;
            pvdd-supply = <&reg_audio_p>;
    };
};

&sai5 {
    status = "okay";
    assigned-clock-parents = <&clk IMX8MM_AUDIO_PLL2_OUT>;
    jassigned-clock-rates = <11289600>;
};

Conclusion

simple-audio-card is a flexible way to describe the audio routing and I strongly prefer this way over write a machine driver for each SoC-CODEC setup.

My example here is kept to a minimum, you probably want to add widgets and routing as well.

simple-audio-card does support rather complex setup with multiple DAI links, amplifier and such. See the device tree bindings [6] for further reading.

Debug kernel with KGDB

Debug kernel with KGDB

What is KGDB?

KGDB intend to be used as a source code level debugger on a running Linux kernel. It works with GDB and allows the user to inspect memory, variables, setup breakpoints, step lines and instructions. Pretty much the same that all application developers are used to, but for the kernel itself.

Allmost every embedded Linux system does have a serial port available, and that is all that you need to connect GDB to your kernel.

One thing to keep in mind, as with all debugging, is that everything that is called timing will be messed up. It will be pretty obvious when you actually pause a running kernel that keeps up the communicates with all hardware. Especially if you have any hardware watchdogs enabled...

Compile the kernel with support for KGDB

There are a few kernel options that you have to enable in order to use KGDB:

  • CONFIG_KGDB to enable remote debugging.
  • CONFIG_KGDB_SERIAL_CONSOLE let you share a serial console with GDB.
  • CONFIG_FRAME_POINTER is used to produce more reliable stack backtraces by inserting code to preserve the frame informations in registers or on the stack.
  • CONFIG_KALLSYMS_ALL to make sure that all symbols are loaded into the kernel image (i.e. symbols from all sections).
  • CONFIG_MAGIC_SYSRQ to be able to make sysrq. More about this below.

KGDBOC

KGDB over console, or kgdboc, let you use a console port as the debugging port. If we only have one serial port available, we could split the console and gdb communication using agent-proxy [2] .

Agent-proxy

To split a serial port into console and GDB communication we could use agent-proxy. Download and compile agent-proxy

git clone http://git.kernel.org/pub/scm/utils/kernel/kgdb/agent-proxy.git
cd agent-proxy
make

Launch agent-proxy

agent-proxy 4440^4441 0 /dev/ttyS0,115200

If your hardware does not support the line break sequence you have to add the -s003 option. You will find out pretty soon if it is needed - if your target continues to run after sending a break, then you should try to add it, in other words:

agent-proxy 4440^4441 0 /dev/ttyS0,115200 -s003

Where ttyS0 is the serial port on your host.

This will create two TCP sockets, one for serial console and with for GDB, which is listening to port 4440 and 4441 respectively.

Connect to the serial console with your favorite client (socat, netcat, telnet...)

telnet localhost 4440

Setup kgdboc with kernel arguments

kgdboc could be used early in the boot process if it is compiled into the kernel as a built-in (not as module), by providing the kgdboc arguments.

Add kgdboc=<tty-device>,[baud] to your command line argument, e.g.

kgdboc=ttyS0,11500

Where ttyS0 is the serial port on the target.

The kgdbwait argument stops the kernel execution and enter the kernel debugger as earliest as possible. This let you connect to the running kernel with GDB.

See kernel parameters [1] for more information.

Setup kgdboc with kernel module

If the kgdb is not compiled to be built-in but as a module, you provide the same arguments while loading the kernel

modprobe kgdboc=ttyS0,115200

Setup kgdboc in runtime using sysfs

It is also possible to enable kgdboc by echoing parameters into sysfs entries

echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc

Connect GDB to a running kernel

Stop execution and wait for debugger

We have to stop the execution of the kernel in order to connect with gdb.

If gdbwait is provided as boot argument, the kernel will stop its execution and wait.

Otherwise we have to trigger this manually by using SysRq-G. This requires that the CONFIG_MAGIC_SYSRQ is enabled in your kernel config.

Your favorite serial application does probably have some keyboard combination to send SYSRQ requests (GNU Screen has "CTRL+A b" for example), otherwise you can use procfs to send the trigger

echo g > /proc/sysrq-trigger

Connect with GDB

Start GDB and provide the vmlinux from your kernel root directory, remember that you have to use the GDB that came with your toolchain. I do always use the -tui flag to start with a nice terminal user interface

aarch64-linux-gnu-gdb -tui ./vmlinux

Now, if you have a separate serial port for gdb, you could connect to it directly

(gdb) set remotebaud 115200
(gdb) target remote /dev/ttyS0

If you are using agent-proxy, then we should connct to port 4441

(gdb) target remote localhost:4441

Now you are able to set breakpoints, watch variables and use gdb as you used to.

/media/kgdb.jpg

One tip is to set a breakpoint at ksys_sync

(gdb) b ksys_sync

This let you run the sync command as trig to go-to-debug-mode.

Raspberry Pi and QEMU

Raspberry Pi and QEMU

What is QEMU?

QEMU is a generic and open source machine emulator and visualizer. It emulates full machines (boards) of different architectures and is useful for both application and kernel development. The CPU itself could be fully emulated (together with devices, memories and so on) or work with a hypervisor such as KVM or Xen.

/media/qemu-logo.png

If support for your hardware is missing, then it is a fairly easy task to write a stub driver that your application can interface. The most fun part for kernel development is so connect GDB (CONFIG_KGDB) to a running kernel, set breakpoints and step through the kernel code.

What about Raspberry Pi and QEMU?

My normal procedure is to build a custom root filesystem with Buildroot [1], clone the kernel source code and build the kernel. This is also my preferred workflow as I have control on what I actually running, but sometimes it could be handy to just take an already built setup and use it.

I'm currently doing some work with the Raspberry Pi 3b+ which is using an Raspbian image, so why not emulate it?

Board support

QEMU provides models of the following Raspberry Pi Boards:

Machine Core Number of cores RAM
raspi0 ARM1176JZF-S 1 512 MiB
raspi1lap ARM1176JZF-S 1 512 MiB
raspi2b Coretx-A7 4 1 GB
raspi3ap Cortex-A53 4 512 MiB
Raspi3b Cortex-A53 4 1 GB

Device support

QEMU provides support for the following devices:

  • ARM1176JZF-S, Cortex-A7 or Cortex-A53 CPU
  • Interrupt controller
  • DMA controller
  • Clock and reset controller (CPRMAN)
  • System Timer
  • GPIO controller
  • Serial ports (BCM2835 AUX - 16550 based - and PL011)
  • Random Number Generator (RNG)
  • Frame Buffer
  • USB host (USBH)
  • GPIO controller
  • SD/MMC host controller
  • SoC thermal sensor
  • USB2 host controller (DWC2 and MPHI)
  • MailBox controller (MBOX)
  • VideoCore firmware (property)

However, it still lacks support for these:

  • Peripheral SPI controller (SPI)
  • Analog to Digital Converter (ADC)
  • Pulse Width Modulation (PWM)

Set it up

Prerequisites

You will need to have qemu-system-aarch64, you could either build it from source [2] or let your Linux distribution install it for you.

If you are using Arch Linux, then you could use pacman

sudo pacman -Sy qemu-system-aarch64

You will also need to do download and extract the Raspian image you want to use

wget https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-09-26/2022-09-22-raspios-bullseye-arm64-lite.img.xz
unxz 2022-09-22-raspios-bullseye-arm64-lite.img.xz

Loopback mount image

The image could be loopback mounted in order to extract the kernel and devicetree. First we need to figure out the first free loopback device

sudo losetup -f
/dev/loop8

Then we could use that device to mount:

sudo losetup /dev/loop8  ./2022-09-22-raspios-bullseye-armhf-lite.img  -P

The -P option force the kernel to scan the partition table. As the sector size of the image is 512 bytes we could omit the --sector-size.

Mount the boot partition and root filesystem

mkdir ./boot ./rootfs
sudo mount /dev/loop8p1 ./boot/
sudo mount /dev/loop8p2 ./rootfs/

Copy kernel and dtb

cp boot/bcm2710-rpi-3-b.dtb .
cp boot/kernel8.img .

If you have any modification you want to do on the root filesystem, do it now before we unmount everything.

sudo umount ./boot/
sudo umount ./rootfs/

Resize image

QEMU requires the image size to be a power of 2, so resize the image to 2GB

qemu-img resize  ./2022-09-22-raspios-bullseye-armhf-lite.img 2G

Very note that this will lose data if you make the image smaller than it currently is

Wrap it up

Everything is now ready for start QEMU. The parameters are quite self-explained

qemu-system-aarch64 \
    -M raspi3b \
    -cpu cortex-a72 \
    -append "rw earlyprintk loglevel=8 console=ttyAMA0,115200 root=/dev/mmcblk0p2 rootdelay=1" \
    -serial stdio \
    -dtb ./bcm2710-rpi-3-b.dtb \
    -sd ./2022-09-22-raspios-bullseye-armhf-lite.img \
    -kernel kernel8.img \
    -m 1G -smp 4

Here we go

raspberrypi login: pi
Password:
Linux raspberrypi 5.10.103-v8+ #1529 SMP PREEMPT Tue Mar 8 12:26:46 GMT 2022 aarch64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
pi@raspberrypi:~$

Conclusion

QEMU is fun. It is a good way to explore stuff like ftrace, step through kernel code or simple run your application on a virtual board. You do not allways have to build everything yourself, sometimes a raspian image could be what you need.

Crosscompile libcamera for RPi

Crosscompile libcamera for RPi

Goal

The goal is to cross-compile libcamera [1] and libcamera-apps [2] for Raspberry Pi using the latest Raspbian [3] (Bullseye) release. Usually you setup the root filesystem with Buildroot [4] or Yocto [5] and generate a SDK that you can use to compile your application. The Raspbian distribution does not come with a SDK so we have to setup our own.

We will use a Raspberry Pi 3b for this.

What is libcamera?

You can read about libcamera in a previous post [6].

Prepare the SD-card

We will boot the Raspberry Pi from a SD-card, so we first have to prepare it.

Download Raspbian Bullseye

You could either download the image [3] yourself or use rpi-imager. Only make sure that you chose the 64-bit version as that is what the toolchain we are going to use is built for.

To download and flash it yourself

wget https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-09-26/2022-09-22-raspios-bullseye-arm64-lite.img.xz
tar xf 2022-09-22-raspios-bullseye-arm64-lite.img.xz
sudo dd if=2022-09-22-raspios-bullseye-arm64-lite.img.xz of=/dev/mmcblk0 conv=sync

/dev/mmcblk0 will be overwritten, please double check that this is your SD-card dedicated for your Raspberry and not your wedding pictures.

I will stick to rpi-imager as it let you configure WLAN, enable SSH and set passwords in the configuration menu. I find it smooth.

/media/rpi-imager.jpg

As I'm using Arch Linux, rpi-imager is available as an AUR package

git clone https://aur.archlinux.org/rpi-imager.git
cd rpi-imager
makepkg -i

Other Debian-based distribution could just use apt-get

apt-get install rpi-imager

Enable UART

Once the image is written to the SD-card we can enable the UART. This is not necessary but I strongly prefer to have a serial port connected to the device I'm working with to see the boot log and get a getty.

Mount the boot partition and enable UART by write "enable_uart=1" to config.txt

sudo mount /dev/mmcblk0p1 mnt/disk/
echo enable_uart=1 | sudo tee -a mnt/disk/config.txt
sudo umount mnt/disk

Permit root login on SSH

Now it is time to power up the Raspberry Pi. Put the SD-card in the slot, power it up and login as the pi user, either via UART or SSH.

Permit root login in order to mount root filesystem via SSH

echo PermitRootLogin yes  | sudo tee -a /etc/ssh/sshd_config

Restart the SSH service

sudo systemctl restart sshd.service

Note that it is bad practice to let the root user login via SSH (especially with password). Either use SSH-keys or disable it later.

If you not allready have figured out which IP address the RPi has, grab it with ip addr

pi@raspberrypi:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:91:e6:2a brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.111/24 brd 192.168.1.255 scope global dynamic noprefixroute eth0
       valid_lft 409sec preferred_lft 334sec
    inet6 fe80::cfe6:1f35:c5b6:aa1a/64 scope link
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:c4:b3:7f brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.56/24 brd 192.168.1.255 scope global dynamic noprefixroute wlan0
       valid_lft 408sec preferred_lft 333sec
    inet6 fe80::61e6:aeb:2c0e:f31b/64 scope link
       valid_lft forever preferred_lft forever

My board has the IP 192.168.1.111 on the eth0 interface.

Set root password

The root user does not have a password by default, so set it

sudo passwd root

Install dependencies

Especially libcamera-apps have a lot of dependencies, install those using apt-get

sudo apt-get install libboost-program-options-dev libboost-dev libexif-dev libjpeg-dev libtiff-dev libpng-dev libdrm-dev libavcodec-dev libavdevice-dev

Prepare the host

Now we have everything on target in place so can switch back to the host system.

From now on, we will use environment variables to setup paths to all directories we will refer to. First, create those directories

mkdir rootfs staging tools

rootfs will be used for our network mounted root filesystem. staging will be our sysroot we compile against tools will contain our cross toolchain

Export the environment variables

export RPI_BASE=`pwd`
export RPI_ROOTFS=$RPI_BASE/rootfs
export RPI_STAGING=$RPI_BASE/staging
export RPI_TOOLS=$RPI_BASE/tools/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu/bin/

Install cross toolchain

Download cross toolchain with the same GCC version (10.2) as Raspian.

wget http://sources.buildroot.net/toolchain-external-arm-aarch64/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz
tar -xf gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz -C tools/

Add the toolchain to your $PATH

export PATH=$RPI_TOOLS:$PATH

Mount root filesystem

SSHFS (Secure SHell File System) [7] is a handy tool based in libfuse that let you mount a filesystem over SSH.

Mount the root filesystem

sshfs root@192.168.1.111:/ $RPI_ROOTFS

Prepare the staging directory

Here we will copy files from our root filesystem

mkdir -p $RPI_STAGING/usr/
cp -r $RPI_ROOTFS/usr/lib $RPI_STAGING/usr/
cp -r $RPI_ROOTFS/usr/include/ $RPI_STAGING/usr/

Now you may use ./rootfs as your sysroot. However, as the Rasbian image has C-library for both musl and glibc, the search pathes becomes hard to handle. We will look on how we could handle this later, but that is simply not worth it in my opinion. Instead, copy the glibc libraries to /usr/lib

cp $RPI_STAGING/usr/lib/aarch64-linux-gnu/* $RPI_STAGING/usr/lib/

We also need to create a symlink to /lib as the toolchain is looking for the linux loader in that directory

ln -s usr/lib/ $RPI_STAGING/lib

The the libpthread.so.6 is pointing to an aboslut path (will point to our host system). The result will be that the linker does not find the library and will fallback on the static linked library instead. That will not fall out good as the glibc is still dynamically linked... so create a new symlink

ln -sf libpthread.so.0 $RPI_STAGING/lib/libpthread.so

Cross compile libcamera

Cross compile libcamera is quite straight forward as it does use Meson as build system and do not have any external dependencies.

Clone the repository

git clone https://git.libcamera.org/libcamera/libcamera.git
cd libcamera

We need to create a toolchain file to instruct the Meson build system which toolchain it should use. Create aarch64.txt which contains

[binaries]
c = 'aarch64-none-linux-gnu-gcc'
cpp = 'aarch64-none-linux-gnu-g++'
ar = 'aarch64-none-linux-gnu-ar'
strip = 'aarch64-none-linux-gnu-strip'

[host_machine]
system = 'linux'
cpu_family = 'x86_64'
cpu = 'x86_64'
endian = 'little'

Now we could build

meson build -Dprefix=/usr/ --cross-file ./aarch64.txt
cd build
ninja

Install on both root filesystem and staging directory

DESTDIR=$RPI_ROOTFS ninja install
DESTDIR=$RPI_STAGING ninja install

Cross compile libcamera-apps

libcamera-apps on the other hand, have a lot of dependencies but as we have collected all of those into the staging directory, even this is pretty straight forward.

Just for your information, if we did not copy the glibc-libraries from /usr/lib/aarch-linux-gnu to /usr/lib, this would not be a straight trip at all.

Clone the repository

git clone https://github.com/raspberrypi/libcamera-apps.git
cd libcamera-apps

I'm not interresting in the preview application as it needs Qt5, so I will remove it

sed -i "/add_subdirectory(preview)/d" CMakeLists.txt

I also did remove libcamera-raw and libcamera-vid from apps/CMakeLists.txt as those have dependencies that I do not want on my target.

libcamera-apps does use CMake as build system, and we need to create a toolchain file for this as well. Create the file aarch64.cmake which contains

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR arm)

set(CMAKE_SYSROOT $ENV{RPI_STAGING})

#We need to point to the glibc-headers
set(CMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES $ENV{RPI_STAGING}/usr/include/aarch64-linux-gnu/)

set(CMAKE_C_COMPILER $ENV{RPI_TOOLS}/aarch64-none-linux-gnu-gcc)
set(CMAKE_CXX_COMPILER $ENV{RPI_TOOLS}/aarch64-none-linux-gnu-g++)

#Let pkg-config look in our root
SET(ENV{PKG_CONFIG_LIBDIR} ${CMAKE_FIND_ROOT_PATH}/lib/pkgconfig/)

set(CMAKE_FIND_ROOT_PATH "${CMAKE_SYSROOT}")
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

Now we are ready to build

mkdir build
cd build
cmake -DCMAKE_TOOLCHAIN_FILE=../aarch64.cmake ../
make

And install

make DESTDIR=$RPI_ROOTFS install

We do only need to install to our root filessytem as it should run on target.

Test on target

We can now run the applications on target and verify that it is our newly built app:

pi@raspberrypi:/usr/local/bin$ LD_LIBRARY_PATH=/usr/local/lib ./libcamera-jpeg  --version
libcamera-apps build: 80f17befef34-intree-dirty 26-11-2022 (15:55:47)
libcamera build: v0.0.2+34-b35f04b3

For the good sake, we could also take an image

pi@raspberrypi:/usr/local/bin$ LD_LIBRARY_PATH=/usr/local/lib ./libcamera-jpeg  -o /tmp/test1.jpg
/media/libcamera-rpi-small.jpg

Now you can disable root login for SSH

sed -i "/PermitRootLogin/d" /etc/ssh/sshd_config
sudo systemctl restart sshd.service

What if...

...we should not copy /usr/lib/aarch64-linux-gnu to /usr/lib but keep it as it is? It would be nice to get rid of the staging directory and only use the root filesystem for cross compiling.

That was my first intention, but it turned out to be really troublesome, even if I got it to work at last.

In short what is needed to be done is:

  • Export PKG_CONFIG_SYSROOT_DIR to $RPI_ROOTFS
  • Export PKG_CONFDIG_PATH to $RPI_ROOTFS/usr/lib/aarch64-linux-gnu/pkgconfig to make it find pkgconfig files
  • Export BOOST_ROOT to $RPI_ROOTFS to give FindBoost.cmake a hint of where to look
  • Export BOOST_LIBRARYDIR to $RPI_ROOTFS/usr/lib/aarch64-linux-gnu
  • Set CMAKE_LIBRARY_PATH to look into $RPI_ROOTFS/usr/lib/aarch64_linux-gnu
  • Set CMAKE_CXX_FLAGS_INIT, CMAKE_C_FLAGS_INIT and CMAKE_EXE_LINKER_FLAGS to search $RPI_ROOTFS/usr/lib/aarch64-linux-gnu for libraries

There are simply too much special and I think it is not worth it.

Conclusion

This sounds like a simple thing, but it actually took quite a while to get it working. Mostly because of the that the Raspian distribution supports both and glibc/musl so that libraries, pkgconfig files and headers end up in non-standard search paths.

I find it quite strange that there are no SDK available for the Raspian images, it would help development against the platform a lot.

What is libcamera and why should you use it?

What is libcamera and why should you use it

Read out a picture from camera

Once in a time, video devices was not that complex. To use a camera back then, your application software could iterated through /dev/video* devices and pick the camera that you want and then immediately start using it. You could query which pixel formats, frame rates, resolutions and all other properties that are supported by the camera. You could even easely change it if you want.

This still works for some cameras, basically every USB camera and most laptop cameras still works that way.

The problem, especially in embedded systems, is that there is no such thing as "the camera" anymore. The camera system is rather a complex pipeline of different image processing nodes that the image data traverse through to be shaped as you want. Even if the result of this pipeline will end up in a video device, you cannot configure things like cropping, resolution etc. directly on that device as you used to. Instead, you have to use the media controller API to configure and link each of these nodes to build up your pipeline.

To show how it may look like; this is a graph that I had in a previous post [3]:

/media/media-ctl-graph.png

What is libcamera?

/media/libcamera-banner.png

This is how libcamera is described on their website [1]

libcamera is an open source camera stack for many platforms with a core userspace library, and support from the Linux kernel APIs and drivers already in place.
It aims to control the complexity of embedded camera hardware by providing an intuitive API and method of separating untrusted vendor code from the open source core.

libcamera aims to encourage the development of new embedded camera applications by limiting the complexity that developers have to deal with.
The interface is designed around the way that modern embedded camera hardware works.

First time I heard about libcamera was on the Embedded Linux Conference 2019 where Jacopo Mondi had a talk [2] about the Public API for the first stable libcamera release. I have been working with cameras in several embedded Linux products and know for sure how complex [3] these little beast could be. The configuration also differ depending on which platform or camera you are using as there is no common way to setup the image pipe. You will soon have special cases for all your platform variants in your application. Which is not what we strive for.

libcamera is trying to solve this by provide one library that takes care of all that complexity for you.

For example, if you want to adjust a simple thing, say contrast, of a IMX219 camera module connected to a Raspberry Pi. To do that without libcamera, you first have to setup a proper image pipeline that takes the camera module, connect it to the several ISP (Image Signal Processing) blocks that your processor offers in order to get the right image format, resolution and so on. Somewhere between all these configuring, you realise that the camera module nor the ISPs have support for adjust the contrast. Too bad. To achieve this you have to take the image, pass it to a self-written contrast algorithm, create a gamma curve that the IPA (Image Processing Algorithm) understands and actually set gamma. Yes, the contrast is adjusted with a gamma curve for that particular camera on Raspberry Pi. ( Have a look at the implementation of that IPA block [7] for Raspberry Pi )

This is exactly the stuff libcamera understands and abstract for the user. libcamera will figure out what graph it has to build depending on what you want do to and which processing operations that are available at your various nodes. The application that is using libcamera for the video device will set contrast for all cameras and platforms. After all, that is what you wanted.

Camera Stack

As the libcamera library is fully implemented in userspace and use already existing kernel interfaces for communication with hardware, you will need no extra underlying support in terms of separate drivers or kernel support.

libcamera itself exposes several API's depending on how the application want to interface the camera. It even have a V4L2 compatiblity layer to emulate a high-level V4L2 camera device to make a smooth transistion for all those V4L2 applications out there.

/media/libcamera-layer.png

Read more about the camera stack in the libcamera documentation [4].

Conclusion

I really like this project and I think we need an open-source stack that supports many platforms. This vendor-specific drivers/libraries/IPAs-situation we are in right now is not sustainable at all. It takes too much effort to evaluate a few cameras of different vendors just because all vendors has their own way to control the camera with their own closed-source and platform specific layers. Been there done that.

For those vendors that do not want to open-source their secret image processing algorithms, libcamera uses a plugin system for IPA modues which let vendors keep their secrets but still be compatible with libcamera. All open-source modules are identified based on digital signatures, while closed-source modules are instead isolated inside a sandbox environment with restricted access to the system. A Win-Win concept.

The project itself is still quite young and need more work to support more platforms and cameras, but the ground is stable. Raspberry Pi is now a common used platform, both in commersial and hobby, and the fact that Raspberry Pi Foundation has choosen libcamera as their primary camera system [8] must tell us something.

HID report descriptors and Linux

HID report descriptors and Linux

HID Devices

USB HID (Human Interface Device) device class is the type of computer perihpherals that human interacts with, such as keyboards, mice, game controllers and touchscreens. The protocol is probably one of the most simple protocols in the USB specification. Even if HID was originally written for USB in mind, it works with several other transport layers. Your mouse and keyboard do probably use HID over USB, the touchscreen in your smartphone could use HID over I2C. Even Bluetooth/BLE make use of the same protool for this type of devices.

The protocol is popular as it is so simple. The HID descriptor could be stored in ROM and the perihpheral could be implemented using only a small 8-bit MCU.

Dispite how simple and well defined [1] the HID specification is, implementors will still get it wrong as we will see later on. Those noncompliant errors has to be fixed up in order to use the device. It is quite sad as it requires a separate driver for this even if the HID should be able to be handled in a generic way.

We will continue to focus on the HID implementation over USB.

HID Descriptors

The device itself has to identify itself and that information is stored in segments of its ROM (Read Only Memory). This segments, or descriptors as they are called, describe what type of device it is and which interface (Interface Descriptor) it exposes. Such interfaces are called classes and all devices belongs to one USB class. Such class [2] could be Printer, Video, or the one we will focus on - the HID class.

/media/hid-1.png

The HID class device descriptor defines and identifies which other descriptors present. Such other descriptors could be report descriptors and physical descriptors.

Report Descriptors

The report descriptor describes how the data that the device generates should be interpreted. For example, the report descriptor describes how to determine the button state of your mouse or the position of the touchscreen click on your smartphone.

Physical Descriptors

The physical descriptor on the other hand provides information about the physical layout of the device, e.g. what and how your body interact with the device to activate certain functions.

The big picture

There are of cause more descriptors, but most of them are part of the USB specification rather than specific for the HID devices.

/media/hid-2.png

There are much to say about the HID class, subclasses, interfaces and protocols and we will not cover them all. But just to give you a hint of what it is:

HID class

USB devices are grouped into USB classes depending on what type of device and transport requirement it is. For example an Audio device requires isohcronous data pipes which HID devices does not. HID Devices has different and much simplier data transport requirements.

Subclasses

The only subclass for the HID class is the Boot Interface Subclass. It is a small subset of the report descriptor that is easier to parse for code that does not want (or have resource to) parse a full report descriptor. BIOS is one example of such code that want to keep the complexity and footprint as small as possible, but still want to be able to use a keyboard.

Protocol

The HID protocol only has a meaning if subclass is a Boot Interface, then it is used to determine if the device is a mouse or keyboard. Otherwise the Protocol is not used.

Interfaces

The interface is a way for the device and host to communicate.HID devices use either the Control pipe or the Interrupt pipe.

Control pipes are used for:

  • Receive and respond to requests for USB control and class data
  • Transmit data when asked by the HID class driver
  • Receive data from the host

The interrupt pipe are used for:

  • Receiving asynchronous data from the device
  • Transmitting low-latency

Report descriptors

The information passed to and from the device is encapsulated in reports. These reports are organized data where the data layout is described in a report descripor. This report descriptor is one of the first items that the host requests from the device and describes how the data should be interpreted. For example, a mouse could have several buttons represented by one bit each and a wheel represented from -127 to +128. The report descriptor will give you details about which bits are mapped to which button and also which 8 bits should be used for the wheel.

All report descriports are available to read out from sysfs. This is the report from my keyboard

[15:15:02]marcus@goliat:~$ od -t x1 -Anone  /sys/bus/usb/devices/3-12.3.2.1/3-12.3.2.1:1.0/0003:045E:00DB.0009/report_descriptor
 05 01 09 06 a1 01 05 08 19 01 29 03 15 00 25 01
 75 01 95 03 91 02 09 4b 95 01 91 02 95 04 91 01
 05 07 19 e0 29 e7 95 08 81 02 75 08 95 01 81 01
 19 00 29 91 26 ff 00 95 06 81 00 c0

Here is a another (parsed) example of what the report descriptor may look like

0x05, 0x01,        // Usage Page (Generic Desktop Ctrls)
0x09, 0x04,        // Usage (Joystick)
0xA1, 0x01,        // Collection (Application)
0xA1, 0x00,        //   Collection (Physical)
0x09, 0x30,        //     Usage (X)
0x09, 0x31,        //     Usage (Y)
0x15, 0x00,        //     Logical Minimum (0)
0x26, 0xFF, 0x07,  //     Logical Maximum (2047)
0x35, 0x00,        //     Physical Minimum (0)
0x46, 0xFF, 0x00,  //     Physical Maximum (255)
0x75, 0x10,        //     Report Size (16)
0x95, 0x02,        //     Report Count (2)
0x81, 0x02,        //     Input (Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position)
0xC0,              //   End Collection
0x75, 0x08,        //   Report Size (8)
0x95, 0x03,        //   Report Count (3)
0x81, 0x03,        //   Input (Cnst,Var,Abs)
0xC0,              // End Collection

The descriptor describes a 2 axis (X/Y) joystick controller where each axis could have an absolute value between 0 and 2047.

If we walk through the descriptor step by step.

0x05, 0x01,        // Usage Page (Generic Desktop Ctrls)

As there are a many different types of devices, these are grouped into pages. This to know how the following (Usage) entry should be intepreted.

There are a plenty of groups as seen in the reference manual [3] :

/media/hid-page.png

Our joystick belongs to the Generic Desktop Ctrls group.

The next line is the usage:

0x09, 0x04,        // Usage (Joystick)

Even mouse, keyboard and gamepads are example of Generic Desktop Ctrls as seen in the table below:

/media/hid-page.png

The next entry is the application collection:

0xA1, 0x01,        // Collection (Application)

The application collection is to make a meaningful grouping of Input, Output and Feature items. Each such grouping has at least Report Size and Report Count defined to determine how big in terms of bytes the collections is.

The next entry is the physical collection:

0xA1, 0x00,        //   Collection (Physical)

This provides information about the part or parts of the human body used to activate the controls on the device. In other words - button and knobs.

Now to the more fun part:

0x09, 0x30,        //     Usage (X)
0x09, 0x31,        //     Usage (Y)
0x15, 0x00,        //     Logical Minimum (0)
0x26, 0xFF, 0x07,  //     Logical Maximum (2047)
0x35, 0x00,        //     Physical Minimum (0)
0x46, 0xFF, 0x00,  //     Physical Maximum (255)
0x75, 0x10,        //     Report Size (16)
0x95, 0x02,        //     Report Count (2)
0x81, 0x02,        //     Input (Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position)

Here we can see that there are two axis, X/Y, that is represented by a value between 0 and 2047 (11 bits). A single axis does ockupy 16 bits and is of the type Input. That is pretty much all what we need to know in order to parse this information.

What is so hard then?

These report descriptors is not that hard to follow and there is no black magic around them. Dispite that, many vendors does not get these report descriptors right and keep deliver a custom driver along with their product.

I used to build and fly tri- and quadcopters, and on way to be better at flying is to use your radio transmitter connected to a simulator as training. The crashes is not that fatal nor costly that way...

I've never seen such a flight controller that actually follow the HID specification, and that makes them useless without a custom driver that can parse the mess. It is not uncommon that the actual reports from the device looks good, it is just that the report descriptor is messed up.

In that case we can write a pretty small Linux Kernel Driver that only fixup the report descriptor and then let the HID layer create and manage the device in a generic way. This is what I did for the VRC2 and HID-PXRC driver [4] which will be available in Linux 6.1.

Such driver could be as simple as (cut out from my VRC2-driver):

static __u8 vrc2_rdesc_fixed[] = {
    0x05, 0x01,        // Usage Page (Generic Desktop Ctrls)
    0x09, 0x04,        // Usage (Joystick)
    0xA1, 0x01,        // Collection (Application)
    0x09, 0x01,        //   Usage (Pointer)
    0xA1, 0x00,        //   Collection (Physical)
    0x09, 0x30,        //     Usage (X)
    0x09, 0x31,        //     Usage (Y)
    0x15, 0x00,        //     Logical Minimum (0)
    0x26, 0xFF, 0x07,  //     Logical Maximum (2047)
    0x35, 0x00,        //     Physical Minimum (0)
    0x46, 0xFF, 0x00,  //     Physical Maximum (255)
    0x75, 0x10,        //     Report Size (16)
    0x95, 0x02,        //     Report Count (2)
    0x81, 0x02,        //     Input (Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position)
    0xC0,              //   End Collection
    0x75, 0x08,        //   Report Size (8)
    0x95, 0x03,        //   Report Count (3)
    0x81, 0x03,        //   Input (Cnst,Var,Abs)
    0xC0,              // End Collection
};

static __u8 *vrc2_report_fixup(struct hid_device *hdev, __u8 *rdesc,
                unsigned int *rsize)
{
    hid_info(hdev, "fixing up VRC-2 report descriptor\n");
    *rsize = sizeof(vrc2_rdesc_fixed);
    return vrc2_rdesc_fixed;
}

static int vrc2_probe(struct hid_device *hdev, const struct hid_device_id *id)
{
    int ret;

    /*
     * The device gives us 2 separate USB endpoints.
     * One of those (the one with report descriptor size of 23) is just bogus so ignore it
     */
    if (hdev->dev_rsize == 23)
        return -ENODEV;

    ret = hid_parse(hdev);
    if (ret) {
        hid_err(hdev, "parse failed\n");
        return ret;
    }

    ret = hid_hw_start(hdev, HID_CONNECT_DEFAULT);
    if (ret) {
        hid_err(hdev, "hw start failed\n");
        return ret;
    }

    return 0;
}

static const struct hid_device_id vrc2_devices[] = {
    { HID_USB_DEVICE(USB_VENDOR_ID_VRC2, USB_DEVICE_ID_VRC2) },
    { /* sentinel */ }
};
MODULE_DEVICE_TABLE(hid, vrc2_devices);

static struct hid_driver vrc2_driver = {
    .name = "vrc2",
    .id_table = vrc2_devices,
    .report_fixup = vrc2_report_fixup,
    .probe = vrc2_probe,
};
module_hid_driver(vrc2_driver);

MODULE_AUTHOR("Marcus Folkesson <marcus.folkesson@gmail.com>");
MODULE_DESCRIPTION("HID driver for VRC-2 2-axis Car controller");
MODULE_LICENSE("GPL");

BPF for HID drivers

Benjamin Tissoires, One of the maintiners for the HID core layer, has posted his work [6] to introduce eBPF (extended Berkely Packet Filter) support for HID devics which is a really cool thing. As many devices just lack a proper report descriptor, the eBPF let you write such fixup in userspace and simply load the program into the kernel. There is still some parts missing before we can see full support for this feture, but the main part is merged and will be available in 6.1.

See the LWN article [5] for further reading.

Conclusion

HID report descriptors has been a fun subject to dig into. It is still hard to see why different vendors has so hard to follow the specification though.

I also have to thank Benjamin Tissoires for great help and support in understanding how the HID layer and HID devices works.

Industrial I/O and triggers

Industrial I/O and triggers

I've maintained a couple of IIO-drivers (MCP3911 [4] and LTC1660 [5]) for some time now and it is time to give at least the MCP3911 a face-lift.

This time the facelift includes support for:

  • Buffers
  • Triggers
  • Make the driver interrupt driven
  • Add support for setting Oversampling Ratio
  • Add support for setting PGA (Pre Gain Amplifier)

Also clean it up a bit by only using device managed resources.

What is Industrial I/O?

Industrial I/O, or IIO [1], is a subsystem that exposes sensors and actuators in a common way to userspace. The subsystem supports a range of different sensors including ADCs, IMUs, pressure sensors, light sensors, accelerometers and more. Even actuators such as DACs and amplifiers has its place in the IIO subsystem.

The hwmon [2] subsystem provides an interface for a few types of sensors as well, but the framework lack support to cover some use cases that IIO tries to solve, such as:

  • High speed sensors
  • Triggered sampling
  • Data buffering

In short, use hwmon for slow sensors and actuators, otherwise use IIO (preferred for new devices).

The IIO subsystem also provides a stable ABI for various userspace HALs which hwmon does not. libiio [3] is the official and preferred one for the IIO.

/media/iio.png

Sysfs

All IIO devices is exported by sysfs where those can be configured and read single shot values. For example, the raw value of the first channel of an ADC can be read out by:

cat /sys/bus/devices/iio:device/in_voltage0_raw

All other parameters such as oversampling ratio and scaling value is also exposed here.

Scaling value

The value you get from in_voltageX_raw is the raw value, it means that it has to be converted in order to get something meaningful out of it.

To get the value in mV you have to take the scale and offset value into account:

Value in mV = (raw + offset) * scale

All these values are exposed by sysfs in in_voltage_scale and in_voltage_offset respectively.

Triggers

Triggers can be both hardware and software based.

Example on hardware based triggers are:

  • GPIO-based interrupts

Example on software based triggers are:

  • sysfs - you can trig a data poll from userspace
  • hrtimer - let you specify the period and a High Resolution timer will be created and trig a data poll at a given frequency

CONFIG_IIO_SYSFS_TRIGGER

By enable CONFIG_IIO_SYSFS_TRIGGER you can make use of the sysfs trigger

# echo 0 > /sys/bus/iio/devices/iio_sysfs_trigger/add_trigger
# cat /sys/bus/iio/devices/iio_sysfs_trigger/trigger0/name
sysfstrig0

CONFIG_IIO_HRTIMER_TRIGGER

By enable CONFIG_IIO_HRTIMER_TRIGGER you can make use of a timer based trigger

# mount -t configfs none /sys/kernel/config
# mkdir /config/iio/triggers/hrtimer/my_50ms_trigger
# echo 2000 > /sys/bus/iio/devices/trigger0/sampling_frequency

Make use of a trigger

As long as the device supports triggers, there will be an /sys/bus/iio/devices/iio:device0/trigger/current_trigger entry. All available triggers, both hardware and software based, are located in /sys/bus/iio/devices/triggerX.

One nice feature is that one trigger can be used for multiple devices.

In order to activate a trigger for a certain device, simply write the trigger name to the current_trigger entry:

# cat /sys/bus/iio/devices/trigger0/name > /sys/bus/iio/devices/iio:device0/trigger/current_trigger

The next step is to decide and enable those channels you want to scan

# echo 1 > /sys/bus/iio/devices/iio:device0/scan_elements/in_voltage0_en
# echo 1 > /sys/bus/iio/devices/iio:device0/scan_elements/in_voltage1_en
# echo 1 > /sys/bus/iio/devices/iio:device0/scan_elements/in_timestamp_en

And finally, start the sampling process

# echo 1 > /sys/bus/iio/devices/iio:device0/buffer/enable

Now you will get the raw values for voltage0, voltage1 and the timestamp by reading from the /dev/iio:device0 device.

You will read out a stream of data from the device. Before applying the scale value to the raw data, the buffer data may be processed somehow depending on its format. The format for each channel is available as a sysfs entry as well:

# cat /sys/bus/iio/devices/iio:device0/scan_elements/in_voltage0_type
be:s24/32>>0

The buffer format for voltage0 means that each sample is 32 bits wide, does not need any shifting and should be intepreted as a signed 24-bit value.

Conclusion

The IIO subsystem is rather complex. The framework also supports events which makes it possible to trig on specific threshold values. As the subsystem is optimized for performance and the triggers makes it possible to read values at a given frequency or event, this makes a lot more use cases possible than the older hwmon interface.

FYI, the patches for MCP3911 is currently up to be merged into mainline.

Mounting with systemd and udev

/media/systemd.png

Mounting with systemd and udev

Systemd has not allways been my first choice as init system for embedded system, but I cannot ignore that it has many good and handy things that other init systems don't. At the same time, that is just what I don't like with systemd, it does not follow the "Do one thing and do it well"-philosophy that I like so much. I am very thorn about it.

However, when trying to do some things with systemd as you used to do with other systems you sometimes encounter some difficulties. Mostly it is simple because there is another way to accomplish what you want, the "systemd-way", which is usually a better and safer way, but sometimes you simply don't want to.

One such thing I encountered was to mount filesystems with udev. This used to work, but since v239 of systemd, two separate directives were introduced and changed this default behavior.

units: switch from system call blacklist to whitelist

Commit ee8f26180d01e3ddd4e5f20b03b81e5e737657ae [1]

units: switch from system call blacklist to whitelist

This is generally the safer approach, and is what container managers
(including nspawn) do, hence let's move to this too for our own
services. This is particularly useful as this this means the new
@System-service system call filter group will get serious real-life
testing quickly.

This also switches from firing SIGSYS on unexpected syscalls to
returning EPERM. This would have probably been a better default anyway,
but it's hard to change that these days. When whitelisting system calls
SIGSYS is highly problematic as system calls that are newly introduced
to Linux become minefields for services otherwise.

Note that this enables a system call filter for udev for the first time,
and will block @clock, @mount and @swap from it. Some downstream
distributions might want to revert this locally if they want to permit
unsafe operations on udev rules, but in general this shiuld be mostly
safe, as we already set MountFlags=shared for udevd, hence at least
@mount won't change anything.

This patch change the default filter behavior from a blacklist to a whitelist and @mount is no longer allowed

+ SystemCallFilter=@system-service @module @raw-io
+ SystemCallErrorNumber=EPERM

units: switch udev service to use PrivateMounts=yes

Commit b2e8ae7380d009ab9f9260a34e251ac5990b01ca [2]

units: switch udev service to use PrivateMounts=yes

Given that PrivateMounts=yes is the "successor" to MountFlags=slave in
unit files, let's make use of it for udevd.

What does systemd says about PrivateMounts? [3]

PrivateMounts=
Takes a boolean parameter. If set, the processes of this unit will be run in their own private file system (mount) namespace with all mount propagation from the processes towards the host's main file system namespace turned off. This means any file system mount points established or removed by the unit's processes will be private to them and not be visible to the host. However, file system mount points established or removed on the host will be propagated to the unit's processes. See mount_namespaces(7) for details on file system namespaces. Defaults to off.

When turned on, this executes three operations for each invoked process: a new CLONE_NEWNS namespace is created, after which all existing mounts are remounted to MS_SLAVE to disable propagation from the unit's processes to the host (but leaving propagation in the opposite direction in effect). Finally, the mounts are remounted again to the propagation mode configured with MountFlags=, see below.

File system namespaces are set up individually for each process forked off by the service manager. Mounts established in the namespace of the process created by ExecStartPre= will hence be cleaned up automatically as soon as that process exits and will not be available to subsequent processes forked off for ExecStart= (and similar applies to the various other commands configured for units). Similarly, JoinsNamespaceOf= does not permit sharing kernel mount namespaces between units, it only enables sharing of the /tmp/ and /var/tmp/ directories.

Other file system namespace unit settings — PrivateMounts=, PrivateTmp=, PrivateDevices=, ProtectSystem=, ProtectHome=, ReadOnlyPaths=, InaccessiblePaths=, ReadWritePaths=, … — also enable file system namespacing in a fashion equivalent to this option. Hence it is primarily useful to explicitly request this behaviour if none of the other settings are used.

This option is only available for system services, or for services running in per-user instances of the service manager when PrivateUsers= is enabled.

If PrivateMounts=true, then the process has its own mount namespace which will result in that the mounted filesystem is visable only for the process (udevd) itself and will not be propagated to the whole system.

Conclusion

There is reasons to not allow udev mount filesystems for sure, but if you still want to do it you have to revert these changes by modify /lib/systemd/system/systemd-udev.service with:

GPLv2 and GPLv3

Open Source

"Free as in freedom - not as in free beer". Free beer is nice, but freedom is even nicer.

I have been working with companies from different sections including consumer electronics, military applications, automotive and aeronautics. One common question, regardless of section, is "Can we really use Open Source in our product?". The answer is usually Yes, you can, but....

One common misunderstanding is to interpret Open Source as in free beer. This is kind of true for some Open Source, but that is nothing you can take for granted. The "rules for how the code may be used is specified by its license.

Of those who think they had understood the difference, there is a common misunderstanding that no Open Source software is free and does not belong in any commercial products. Both misunderstandings are of course wrong, but you have to make sure that you understand the licenses you are using.

Before you start to work with any source code (not only Open Source) you always have to take the license into consideration. If do your homework you can avoid surprises and practical implications that otherwise can cause you a delayed project or legal inconveniences.

In short, you have to know what you are doing, and that should not differ from other parts of your development.

Open Source Licenses

"Open source licenses are licenses that comply with the Open Source Definition — in brief, they allow software to be freely used, modified, and shared. To be approved by the Open Source Initiative (also known as the OSI), a license must go through the Open Source Initiative's license review process."

This text is taken from the Open Source Initiative webpage [4], which is an organization that works with defining criterea for Open Source and certificate licenses that comply with OSD (Open Source Definition).

Open Source Definition

Many licenses [5] are certified and may have different requirements for its users, but they all comply with these "rules":

Free Redistribution

The license shall not restrict any party from selling or giving away the software as a component of an aggregate software distribution containing programs from several different sources. The license shall not require a royalty or other fee for such sale.

Source Code

The program must include source code, and must allow distribution in source code as well as compiled form. Where some form of a product is not distributed with source code, there must be a well-publicized means of obtaining the source code for no more than a reasonable reproduction cost, preferably downloading via the Internet without charge. The source code must be the preferred form in which a programmer would modify the program. Deliberately obfuscated source code is not allowed. Intermediate forms such as the output of a preprocessor or translator are not allowed.

Derived Works

The license must allow modifications and derived works, and must allow them to be distributed under the same terms as the license of the original software.

Integrity of The Author's Source Code

The license may restrict source-code from being distributed in modified form only if the license allows the distribution of "patch files" with the source code for the purpose of modifying the program at build time. The license must explicitly permit distribution of software built from modified source code. The license may require derived works to carry a different name or version number from the original software.

No Discrimination Against Persons or Groups

The license must not discriminate against any person or group of persons.

No Discrimination Against Fields of Endeavor

The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.

Distribution of License

The rights attached to the program must apply to all to whom the program is redistributed without the need for execution of an additional license by those parties.

License Must Not Be Specific to a Product

The rights attached to the program must not depend on the program's being part of a particular software distribution. If the program is extracted from that distribution and used or distributed within the terms of the program's license, all parties to whom the program is redistributed should have the same rights as those that are granted in conjunction with the original software distribution.

License Must Not Restrict Other Software

The license must not place restrictions on other software that is distributed along with the licensed software. For example, the license must not insist that all other programs distributed on the same medium must be open-source software.

License Must Be Technology-Neutral

No provision of the license may be predicated on any individual technology or style of interface.

GPL

GPL, or General Public License, is one of the most common Open Source Licenses you will find out there. At least version 2, GPLv2, is something you will encounter for sure if you intend to build an embedded Linux system as the kernel [6] itself is using this license.

GPLv2

So what do you need to comply with GPLv2 code? Basically, you need to provide the source code for all GPLv2 licensed code. Yes, that includes all your modifications too, and this part could seem scary at the first glare.

But will you need to make any changes? Probably. If you want to run Linux on your system you will probably have to make some adaptions to the Linux kernel specific for your board, those changes will follow the GPLv2 license and should be provided as well.

The license is stated as follows:

"The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. "

GPLv2, Sec.3 [2]

If any of those changes is your top-secret algorithm then you have done it wrong by design anyway. Please note that no installation information is required at all which makes it more sufficient for embedded devices.

Tivoization

Tivoization, TiVO ran GPLv2 only Linux kernel, but had HW signature keys that made it possible to only run signed kernels. Even if the TiVO did provide the kernel code, the TiVO customers could not build and install the firmware.

The Free Software Foundation(FSF) found this objectionable as it violates one of the purposes the GPLv2 license had. So FSF ended up with GPLv3 to solve this.

GPLv3

/media/gplv3.png

(Yes, this logo is under Public Domain [7] )

One big difference between v2 and v3 is this part

" “Installation Information” for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. "

GPLv3, Sec .6 [1]

Which states that "Installation information" must be provided together with the source code, in short, you have to provide instruction for an end-user to build and replace GPLv3 parts of your product. But there are also a few exceptions. Most of them is more or less hard to make any use of in a real world product.

Exception 1

It only requred for "User Products". It is hard to say if this is an exception or not as most of the products that use GPLv3 is user products. But the license states that it only affects User products. Consult your lawyer as it is not entirely clear what a "User product" really are.

Exception 2

Only if device is sold or offered for long-term lease. As with all lega stuff, things are a bit unclear. Does remote access or temporary possessions qualify for example?

Please note that even long term lease need you to provide installation information.

Exception 3

If non-modifiable by anyone, you don't have to give the information how to reinstalling or modify binaries.

If you want to be able to update your software, then you will probably need to provide the "installation information".

Exception 4

You can void any warranties if binaries are modified, but you can't prevent reinstallation of modigied binaries.

Conclusion

There is a reason why an author of code chose to use any particular license, and it is important (both for principal and legal reasons) to respect that. Some licenses are more or less appropriate for specific products, but the general rule I follow is to avoid any GPLv3 (both GPLv3 and LGPLv3) licensed software in any embedded system as it hard to be fully compliant. The installation information is often something that companies want to keep for themself, with all rights.

What is my opinion about this? Well, I do like to have the freedom to install whatever software I want in the products I own, but there are circumstances where I'm not sure if it is a good idea when it comes to safety and liability. If I buy a car in second hand, I don't want the software for my airbag or braking system to be "fixed" by some random guy. I think that the Open Source has limited use in too restrictive licenses, and that is somehow counterproductive for the Open Source itself.