OCOTP Layout

The fuses related to MAC addresses starts at offset 0x640 for iMX7 and iMX8MP, and at offset 0x620 for all iMX6 processors.

The MAC fuses belongs to fuse bank 9 as seen in the table below:

Burn fuses

There are several ways to burn the fuses nowadays. A few years ago, the only way (that I'm aware of) was by the non-mainlined fsl_otp-driver provided in the Freescale kernel tree. I'm not going to describe how to use since it should not be used anyway.

The fuses are mapped to the MAC address as described in this picture:

The iMX8MP has two MACs and we will assign the MAC address 00:bb:cc:dd:ee:ff for MAC0 and 00:22:33:44:55:66 for MAC1.

u-boot=> fuse
fuse - Fuse sub-system

Usage:
fuse read <bank> <word> [<cnt>] - read 1 or 'cnt' fuse words,
    starting at 'word'
fuse sense <bank> <word> [<cnt>] - sense 1 or 'cnt' fuse words,
    starting at 'word'
fuse prog [-y] <bank> <word> <hexval> [<hexval>...] - program 1 or
    several fuse words, starting at 'word' (PERMANENT)
fuse override <bank> <word> <hexval> [<hexval>...] - override 1 or
    several fuse words, starting at 'word'

fuse prog -y 9 0 0xccddeeff
fuse prog -y 9 1 0x556600bb
fuse prog -y 9 2 0x00223344

u-boot=> fuse sense 9 0
Sensing bank 9:

Word 0x00000000: ccddeeff
u-boot=> fuse sense 9 1
Sensing bank 9:

Word 0x00000001: 556600bb
u-boot=> fuse sense 9 2
Sensing bank 9:

Word 0x00000002: 00223344

$ cat imx8mplus-emmc-all.uuu 
uuu_version 1.2.39

# This script will flash u-boot to mmc on bus 1
# Usage: uuu <script>

SDPS: boot -f ../imx-boot

#Burn fuses
FB: ucmd fuse prog -y 9 0 0xccddeeff
FB: ucmd fuse prog -y 9 1 0x556600bb
FB: ucmd fuse prog -y 9 2 0x00223344

#Burn image
FB: ucmd setenv emmc_dev 2
FB: ucmd setenv emmc_ack 1
FB: ucmd setenv fastboot_dev mmc
FB: ucmd setenv mmcdev ${emmc_dev}
FB: ucmd mmc dev ${emmc_dev}
FB: flash -raw2sparse all ../distro-image-dev-imx8mp.wic
FB: ucmd mmc partconf ${emmc_dev} ${emmc_ack} 1 0
FB: done

root@imx8mp:~# hexdump /sys/devices/platform/soc@0/30000000.bus/30350000.efuse/imx-ocotp0/nvmem 
0000000 a9eb ffaf aaff 0002 52bb ea35 6000 1119
0000010 4591 2002 0000 0100 007f 0000 2000 9800
0000020 0000 0000 0000 0000 0000 0000 0000 0000
*
0000040 bada bada bada bada bada bada bada bada
*
0000060 0000 0000 0000 0000 0000 0000 0000 0000
*
0000080 0000 0000 0000 0000 0000 0000 0004 0000
0000090 eeff ccdd 00bb 5566 3344 0022 0000 0000

root@imx8mp:~# ip a
4: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
5: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:22:33:44:55:66 brd ff:ff:ff:ff:ff:ff

References

Hands on

To do this, all you need is to have support for network namespaces in the kernel (CONFIG_NET_NS=y) and the iproute2 [1] package, which both probably is included in every standard Linux distribution nowadays.

We will create two network namespaces, lets call them netns_eth0 and netns_eth1:

ip netns add netns_eth0
ip netns add netns_eth1

Move each adapter to their new home:

ip link set eth0 netns netns_eth0
ip link set eth1 netns netns_eth1

Assign ip addresses:

ip netns exec netns_eth0 ip addr add dev eth0 192.168.0.1/24
ip netns exec netns_eth1 ip addr add dev eth1 192.168.0.2/24

Bring up the interfaces:

ip netns exec netns_eth0 ip link set eth0 up
ip netns exec netns_eth1 ip link set eth1 up

Now we can ping each interface and know for sure that the traffic is actually on the cable:

ip netns exec netns_eth0 ping 192.168.0.2
ip netns exec netns_eth1 ping 192.168.0.1

References

Kernel configuration

CRIU makes use of a lot of features that has to be enabled in the Linux kernel for full usage.

CONFIG_CHECKPOINT_RESTORE will be set by the package itself, but there are more configuration options that could be useful depending on how you intend to use the tool.

Relevant configuration options are:

General setup options

• CONFIG_CHECKPOINT_RESTORE=y (Checkpoint/restore support)
• CONFIG_NAMESPACES=y (Namespaces support)
• CONFIG_UTS_NS=y (Namespaces support -> UTS namespace)
• CONFIG_IPC_NS=y (Namespaces support -> IPC namespace)
• CONFIG_SYSVIPC_SYSCTL=y
• CONFIG_PID_NS=y (Namespaces support -> PID namespaces)
• CONFIG_NET_NS=y (Namespaces support -> Network namespace)
• CONFIG_FHANDLE=y (Open by fhandle syscalls)
• CONFIG_EVENTFD=y (Enable eventfd() system call)
• CONFIG_EPOLL=y (Enable eventpoll support)
• CONFIG_RSEQ=y (Enable eventpoll support)

Networking support -> Networking options options for sock-diag subsystem

• CONFIG_UNIX_DIAG=y (Unix domain sockets -> UNIX: socket monitoring interface)
• CONFIG_INET_DIAG=y (TCP/IP networking -> INET: socket monitoring interface)
• CONFIG_INET_UDP_DIAG=y (TCP/IP networking -> INET: socket monitoring interface -> UDP: socket monitoring interface)
• CONFIG_PACKET_DIAG=y (Packet socket -> Packet: sockets monitoring interface)
• CONFIG_NETLINK_DIAG=y (Netlink socket -> Netlink: sockets monitoring interface)
• CONFIG_NETFILTER_XT_MARK=y (Networking support -> Networking options -> Network packet filtering framework (Netfilter) -> Core Netfilter Configuration -> Netfilter Xtables support (required for ip_tables) -> nfmark target and match support)
• CONFIG_TUN=y (Networking support -> Universal TUN/TAP device driver support)

In the beginning of the project, CRIU had their own custom kernel which contained some experimental CRIU related patches. Nowadays many of those patches has been mainlined.

One such patch [3] that I missed in my current kernel verson (v5.10) was introduced in v5.12. It was related to how CRIU gets the process state and is essential to create a checkpoint of a running process:

commit 90f093fa8ea48e5d991332cee160b761423d55c1
Author: Piotr Figiel <figiel@google.com>
Date:   Fri Feb 26 14:51:56 2021 +0100

    rseq, ptrace: Add PTRACE_GET_RSEQ_CONFIGURATION request

    For userspace checkpoint and restore (C/R) a way of getting process state
    containing RSEQ configuration is needed.

    There are two ways this information is going to be used:
     - to re-enable RSEQ for threads which had it enabled before C/R
     - to detect if a thread was in a critical section during C/R

    Since C/R preserves TLS memory and addresses RSEQ ABI will be restored
    using the address registered before C/R.

    Detection whether the thread is in a critical section during C/R is needed
    to enforce behavior of RSEQ abort during C/R. Attaching with ptrace()
    before registers are dumped itself doesn't cause RSEQ abort.
    Restoring the instruction pointer within the critical section is
    problematic because rseq_cs may get cleared before the control is passed
    to the migrated application code leading to RSEQ invariants not being
    preserved. C/R code will use RSEQ ABI address to find the abort handler
    to which the instruction pointer needs to be set.

    To achieve above goals expose the RSEQ ABI address and the signature value
    with the new ptrace request PTRACE_GET_RSEQ_CONFIGURATION.

    This new ptrace request can also be used by debuggers so they are aware
    of stops within restartable sequences in progress.

    Signed-off-by: Piotr Figiel <figiel@google.com>
    Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
    Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
    Reviewed-by: Michal Miroslaw <emmir@google.com>
    Reviewed-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
    Acked-by: Oleg Nesterov <oleg@redhat.com>
    Link: https://lkml.kernel.org/r/20210226135156.1081606-1-figiel@google.com

With that said, to make use of CRIU with the latest features it is highly recommended to use a recent kernel.

And soon it will be available as a package in Buildroot.

References

CRIU

CRIU is a project that implements checkpoint/restore functionality by freeze the state of the process and its sub tasks. CRIU makes use of ptrace [4] to stop the process by attach to the process by sending a PTRACE_SEIZE request. Then it injects parasitic code to dump the process's memory pages into image files to create a recoverable checkpoint.

Such process information is memory pages (collected from /proc/$PID/smaps, /proc/$PID/mapfiles/ and /proc/$PID/pagemap), but also information about opened files, credentials, registers, task states and more.

My first concern was that this could not work very well, how about open sockets (especially clients)? It turns out that CRIU alredy handle most of that stuff. There are only a few scenarios that cannot be dumped [5] yet.

PTRACE

For those not familiar with ptrace(2):

The  ptrace() system call provides a means by which one process (the "tracer") may observe and control the execution of an‐
other process (the "tracee"), and examine and change the tracee's memory and registers.  It is primarily used to  implement
breakpoint debugging and system call tracing.

ptrace is the only interface that the Linux kernel provides to poke around and fetch information from inside another application (think debugger and/or tracers).

The PTRACE_SEIZE was introduced in Linux 3.4:

PTRACE_SEIZE (since Linux 3.4)
       Attach  to  the  process  specified  in  pid,  making  it  a  tracee  of the calling process.  Unlike PTRACE_ATTACH,
       PTRACE_SEIZE does not stop the process.  Group-stops are reported as PTRACE_EVENT_STOP and WSTOPSIG(status)  returns
       the  stop  signal.  Automatically attached children stop with PTRACE_EVENT_STOP and WSTOPSIG(status) returns SIGTRAP
       instead of having SIGSTOP signal delivered  to  them.   execve(2)  does  not  deliver  an  extra  SIGTRAP.   Only  a
       PTRACE_SEIZEd  process can accept PTRACE_INTERRUPT and PTRACE_LISTEN commands.  The "seized" behavior just described
       is inherited by  children  that  are  automatically  attached  using  PTRACE_O_TRACEFORK,  PTRACE_O_TRACEVFORK,  and
       PTRACE_O_TRACECLONE.  addr must be zero.  data contains a bit mask of ptrace options to activate immediately.

       Permission to perform a PTRACE_SEIZE is governed by a ptrace access mode PTRACE_MODE_ATTACH_REALCREDS check; see be‐
       low.

But it took a while until the checkpoint/restore capability was created for this purpose, see capabilities(7):

CAP_CHECKPOINT_RESTORE (since Linux 5.9)
       •  Update /proc/sys/kernel/ns_last_pid (see pid_namespaces(7));
       •  employ the set_tid feature of clone3(2);
       •  read the contents of the symbolic links in /proc/pid/map_files for other processes.

       This capability was added in Linux  5.9  to  separate  out  checkpoint/restore  functionality  from  the  overloaded
       CAP_SYS_ADMIN capability.

Example

I wrote a simple C application that just count a variable up each second and print the value:

    #include <stdio.h>
    #include <unistd.h>
    int main()
    {
        printf("My PID is %i\n", getpid());
        int count = 0;
        while (1) {
            printf("%d\n", count++);
            sleep(1);
        }
    }

Compile the code:

    gcc main.c -o main

Start The application:

    [17:26:03]marcus@goliat:~/tmp/count$ ./main 
    My PID is 2483855
    0
    1
    2
    3
    4
    5
    6

The process is started with process ID 2483855.

We can now dump the process and store its state. We have to add the --shell-job flag to tell that it was spawned from a shell (and therefor have some file descriptors open to PTYs that needs to be restored).

    [17:27:26]marcus@goliat:~/tmp/criu$ sudo criu dump -t 2483855 --shell-job
    Warn  (compel/arch/x86/src/lib/infect.c:356): Will restore 2483855 with interrupted system call

CRIU needs to have the CAP_SYS_ADMIN or the CAP_CHECKPOINT_RESTORE capability. Set it by:

    setcap cap_checkpoint_restore+eip /usr/bin/criu

The criu dump command will now generate a bunch of files to store the current state of the application. These includes open file descriptors, registers, stackframes, memorymaps and more:

    [17:28:00]marcus@goliat:~/tmp/criu$ ls -1
    core-2483855.img
    fdinfo-2.img
    files.img
    fs-2483855.img
    ids-2483855.img
    inventory.img
    mm-2483855.img
    pagemap-2483855.img
    pages-1.img
    pstree.img
    seccomp.img
    stats-dump
    timens-0.img
    tty-info.img

We can now restore the application from where we stopped:

    [17:29:07]marcus@goliat:~/tmp/criu$ sudo criu restore --shell-job
    27
    28
    29
    30

This is cool. But what is even cooler is that you may restore the application on a different host(!).

Why does this exists?

Having a read-only root file system is useful for many scenarios:

• Separate user specific changes from system configuration, and being able to find differences
• Allow factory reset, by deleting the user specific changes
• Have a fallback image in case the user specific changes made the root file system no longer bootable.

Because some data on the root file system changes on first boot or while the system is running, just mounting the complete root file system as read-only breaks many applications. There are different solutions to this problem:

• Symlinking/Bind mounting files and directories that could potentially change while the system is running to a writable partition
• Instead of having a read-only root files system, mounting a writable overlay root file system, that uses a read-only file system as its base and writes changed data to another writable partition.

To implement the first solution, the developer needs to analyse which file needs to change and then create symlinks for them. When doing factory reset, the developer needs to overwrite every file that is linked with the factory configuration, to avoid dangling symlinks/binds. While this is more work on the developer side, it might increase the security, because only files that are symlinked/bind-mounted can be changed.

This meta-layer provides the second solution. Here no investigation of writable files are needed and factory reset can be done by just deleting all files or formatting the writable volume.

How does it work?

The implementation make use of OverlayFS [3], which is a union mount filesystem that combines multiple underlying mount points into one. The filesystem make use of the terms upper and lower filesystem where the upper is filesystem is applied as an overlay on the lower filesystem.

The resulting merge directory is a combination of these two where all files in the upper filesystem overrides all files in the lower.

Dependencies

This layer only depends on:

URI: git://git.openembedded.org/bitbake
branch: kirkstone

and

URI: git://git.openembedded.org/openembedded-core
layers: meta
branch: kirkstone

Usage

BBLAYERS ?= " \
  /path/to/layers/meta \
  /path/to/layers/meta-poky \
  /path/to/layers/meta-yocto-bsp \
  /path/to/layers/meta-readonly-rootfs-overlay \
  "

IMAGE_INSTALL:append = " initscripts-readonly-rootfs-overlay"

root=/dev/sda1 rootrw=/dev/sda2

root=/dev/sda1 rootrw=/dev/sda2 init=/bin/sh

root=/dev/sda1 rootrw=/dev/sda2 init=/init

root=/dev/sda1 rootrw=/dev/sda2 init=/init rootinit=/bin/sh

Details

All kernel parameters that is used to configure meta-readonly-rootfs-overlay:

• root - specifies the read-only root file system device. If this is not specified, the current rootfs is used.
• `rootfstype if support for the read-only file system is not build into the kernel, you can specify the required module name here. It will also be used in the mount command.
• rootoptions specifies the mount options of the read-only file system. Defaults to noatime,nodiratime.
• rootinit if the init parameter was used to specify this init script, rootinit can be used to overwrite the default (/sbin/init).
• rootrw specifies the read-write file system device. If this is not specified, tmpfs is used.
• rootrwfstype if support for the read-write file system is not build into the kernel, you can specify the required module name here. It will also be used in the mount command.
• rootrwoptions specifies the mount options of the read-write file system. Defaults to rw,noatime,mode=755.
• rootrwreset set to yes if you want to delete all the files in the read-write file system prior to building the overlay root files system.

9 Years in the making, the story of Zephyr [1]

Much has happened since the project started by an announcement at an internal event at Intel in 2014. Two years later it went public and was quickly picked up by the Linux Foundation, and now it is listed as one of the top critical open source projects by Google.

Now, in June 2023, it has mad 40 releases and has over a milion lines of code. What a trip, hue?

The project has made a huge progress, but the road has not been straight forward. Many design decisions has been made and changed over time. Not only technical decisions but in all areas. For example, Zephyr was originally BSD licenced. The current license, Apache2, was not the first choice. The license was changed upon requests from other vendors. I think it is good that not only one company has full dominance on the project.

Even the name has been up for discussion before it landed in Zephyr. One fun thing is that Zephyr has completely taken over all search results, it is hard to find anything that are not related to the Zephyr project as it masks out all other hits... oopsie.

Some major transitions and transformations made by the project:

• The build system which was initially a bunch of custom made Makefiles, which then became Kbuild and finally CMake.
• The kernel itself moved from a nano/micro kernel model to a unified kernel.
• Even the review system has changed from Garrit to Github.

The change from the dual kernel model to a unified kernel was made in 2016. The motivation was that the older model suffers from a few drawbacks:

• Non-intutive nature of the nano/micro kernel split
• Double context switch affecting the performance
• Duplication of object types for nano and micro
• System initialixation in the idle task

Instead, we ended up with something that:

• Made the nanokernel 'pre-emptible thread' aware
• Unified fibers and tasks as one type of threads by dropping the Microkernel server
• Allowed cooperative threads to operate on all types of objects
• Clarified duplicated object types
• Created a new, more streamlined API, without any loss of functionality

Many things points to that Zephyr has healthy eco system. If we look at the contributions we can se that the member/ community contributions are strictly increasing every year and the commits by Intel is decreasing.

It shows us that the project itself is an evolving and become more and more of a self- sustaining open eco-system.

System device trees [2]

As the current usage of device tree does not scale well, especially when working with Multi-core AMP SoCs. we have to come up with some alternatives.

One such alternative is the System Device Tree. It is an extenstion of the DT specification that are devleoped in the open. To me it sounded uncomfortible at the first glance, but the talker made it clear that the work is heavily in cooperate with the DT specifications and the Linux device tree maintainters.

The main problem is that there are one instance of everything that is available for all CPUs and that is not suitable for AMP architectures where each core could be of a completely different types. The CPU cores are normally instantiated by one CPU node. One thing that the system device trees contribute to is to change that to independent CPU clusters instead.

Also, in a normal setup, many peripherals are attached to the global simple bus, and are shared across cores. The new indirect-bus on the other hand, which are introduced in System Device Tree, addresses this problem by map the bus to a particular CPU cluster which makes the peripheral visable for a specific set of cores.

System Device Tree will also introduce independent execution domains, of course also mapped to a specific set of CPU cluster. By this we can encapsulate which peripherals that should be accessable from which application.

But how does it work? The suggestion is to let a tool, sysbuild to postprocess the standard DT structure into several standard devicetrees, one for each execution domain.

Manifests: Project sanity in the ever-changing Zephyr world [3]

Mike Szczys talked about manifests files and why you should use those in your project.

But first, what is a manifest file?

It is a file that manages the project hiearchy by specify all repositories by URL, which branch/tag/hash to use and the local path for checkout. The manifest file also support some more advanced features such as:

• Inheritance
• Allow/block lists
• Grouping
• West support for validation

The Zephyr tree already uses Manifest files to manage versions of modules and libraries, and there is no reason to do not use the same method in your application. It let you keep control of which versions of all modules that your application requires in a clear way. Besides, as the manifest file is part of your application repository, it does also has a commit history and all changes to the manifest is trackable and hopefully explained in the commit message.

The inheritance feature in the manifest file is a powerful tool. It let you to import other manifest files and explicitely allow or exclude parts of it. This let you reduce the size of of your project significally.

West will handle everything for you. It will parse the manifest file, recursively clone all repositories and update those to a certain commit/tag/branch. It is preferred to not use branches (or even tags) in the manifest files as those may change. Use the hash if possible. Generally speaking, this is the preferred way in any such system (Yocto, Buildroot, ...).

The biggest benifit that I see is that you treat all dependencies aside from your application and that those dependencies are locked to known versions. Zephyr itself will be treated as a dependency to your application, not the other way around.

It is easy to draw parallells to the Yocto project. My first impression of Yocto was that it is REALLY hard to maintain, pretty much for the same reason that we are talking about here - how do I keep track of every layer in a controllable way? The solution for me wasto use KAS which pretty much do exactly the same thing - it creates a manifest files with all layers (read dependencies) that you can version control.

Zbus [4]

Rodrigo Peixoto, the maintainer and author of the Zbus subsystem had a talk where he gave us an introduction on what it is.

(Rodrigo is a nice guy. If you see him, throw a snowball at him and say hi from me - he will understand).

Zephyr has support for many IPC mechanisms such as LIFO, FIFO, Stack, Message Queue, Mailbox and pipes. All of those works great for one-to-one communication, but that is not allways what we need. Even one-to-many could be tricky with the existing mechanism that Zephyr provides.

ZBus is an internal bus used in Zephyr for Many-to-Many communication, besides, such a infrastructure cover all cases (1:1, 1:N, N:M) as a bonus.

I like these kind of infrastructure. It reminds me of dbus (and kbus..) but in a more simplier manner (and that is a good thing). It allows you to have a event-driven architecture in your application and a unified way to make threads talk and share data. Testability is also a bulletpoint for ZBus. You may easily swap a real sensor for stubbed code and the rest of the system would not notice.

The conference

(I got stuck on a picture. Don't know which talk, but it seems like I enjoyed it)

Brief

It is not an uncommon scenario that a Linux system has several network interfaces that are all up and routeable. For example, consider a laptop with both Ethernet and WiFi.

But how does the system determine which route to use when trying to reach another host?

I was up to setup a system with both a 4G modem and a WiFi connection. My use case was that when the WiFi is available, that interface should be prioritized over 4G. This achieved by adjusting the route metric values for those interfaces.

Metric values

The metric value is one of many fields in the routing table and indicates the cost of the route. This become useful if multiple routes exists to a given destination and the system has to make a decision on which route to use. With that said, the lower metric value (lower cost) a route have, the highter priority i gets.

It is up to you or your network manager to set proper metric values for your routes. The actual value could be determine based on several different factors depending on what is important for your setup. E.g:

• Hop count - The number of routes (hops) in a path to reach a certein network. This is a common metric.
• Delay - Some interfaces have higher delays than others. Compare a 4G modem with a fiber connecton.
• Throughput - The expected throughput of the route.
• Reliability - If some links are more prone på link failures than others, prefer to use other interfaces.

The ip route command will show you all the routes that your system currently have, the last number in the output is the metric value:

$ ip route
default via 192.168.20.1 dev enp0s13f0u1u4 proto dhcp src 192.168.20.173 metric 100
default via 192.168.20.1 dev wlp0s20f3 proto dhcp src 192.168.20.197 metric 600

I have two routes that both is routed via 192.168.20.1.

As you can see, my wlp0s20f3 (Wireless) interface has a higher metric value than my enp0s13f0u1u4 (Ethernet) interface, which will cause the system to choose the ethernet interface over WiFi. In my case, these values are chosen by NetworkManager.

Set metric value

If you want to set specific metric values for your routes, the way will differ depending on how your routes are created.

$ ip route replace default via {IP} dev {DEVICE} metric {METRIC}

$ ifmetric INTERFACE [METRIC]

metric metric
Metrics are used to prefer an interface over another one, lowest wins.

interface wlan0
metric 200

    $ nmcli connection edit tuxnet

    ===| nmcli interactive connection editor | ===

    Editing existing '802-11-wireless' connection: 'tuxnet'

    Type 'help' or '?' for available commands.
    Type 'print' to show all the connection properties.
    Type 'describe [<setting>.<prop>]' for detailed property description.

    You may edit the following settings: connection, 802-11-wireless (wifi), 802-11-wireless-security (wifi-sec), 802-1x, ethtool, match, ipv4, ipv6, tc, proxy
    nmcli> set ipv4.route-metric 600
    nmcli> save
    nmcli> quit

defaultroute
       Add a default route to the system routing tables, using
       the peer as the gateway, when IPCP negotiation is
       successfully completed.  This entry is removed when the
       PPP connection is broken.  This option is privileged if
       the nodefaultroute option has been specified.

defaultroute-metric
       Define the metric of the defaultroute and only add it if
       there is no other default route with the same metric.
       With the default value of -1, the route is only added if
       there is no default route at all.

replacedefaultroute
       This option is a flag to the defaultroute option. If
       defaultroute is set and this flag is also set, pppd
       replaces an existing default route with the new default
       route.  This option is privileged.

replacedefaultroute
defaultroute-metric 900

Summary

It is not that often you actually have to set the metric value yourself. The network manager usually does a great job.

In my system, the NetworkManager did not manage the PPP interface so its metric-logic did not apply to that interface. Therefor I had to let pppd create a default route with a fixed metric.

References

The conference

Lund Linux Conference (LLC) [1] is a "half-open" conference located in Lund. It is a conference with with high quality and I appreciate that the athmosphere is more familiar than at the larger conferences. I've been at the conference a couple of times before and the quality on the talks this year was as good as usual. ( The talks are by the way availalble on Youtube [3].)

We are growing though. Axis generously assists with premisses, but it remains to be seen wether we will get place next year.

Anyway, I took some notes as usual, and this blog post is nothing more than the notes I took during the talk.

Summary

This was far from all talks, but only those that I had some taken some meaningful notes from.

Hope to see you there next year!

References

Brief

Many embedded Linux systems does have some kind of sensitive information on a file storage. It could be private keys, passwords or whatever. It is always a risk that this information could be revealed by an unauthorized person that got their physical hands on the device. The only protection against attackers that who simply bypass the system and access the data storage directly is encryption.

Let's say that we encrypt our sensitive data. Where should we then store the decryption key?

We need to store even that sensitive key on a secure place.

Device mapper

Device mapper is a framework that adds an extra abstraction layer on block devices that lets you create virtual block devices to offer additional features. Such features could be snapshots, RAID, or as in our case, disc encryption.

As you can see in the picture below, the device mapper is a layer in between the Block layer and the Virtual File System (VFS) layer:

The Linux kernel does support a bunch of different mappers. The current kernel (v6.2) does support the following mappers [1]:

• dm-delay
• dm-clone
• dm-crypt
• dm-dust
• dm-ebs
• dm-flakey
• dm-ima
• dm-integrity
• dm-io
• dm-queue-length
• dm-raid
• dm-service-time
• dm-zoned
• dm-era
• dm-linear
• dm-log-writes
• dm-stripe
• dm-switch
• dm-verity
• dm-zero

Where dm-crypt [2] is the one we will focus on. One cool feature of device mappers is that those are stackable. You could for example use dm-crypt on top of a dm-raid mapping. How cool isn't that?

How to use it?

git clone https://github.com/nxp-imx/keyctl_caam.git
cd keyctl_caam

CC=aarch64-linux-gnu-gcc make

./environment-setup-aarch64-poky-linux
make

CONFIG_BLK_DEV_DM=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD=y
CONFIG_DM_CRYPT=y
CONFIG_DM_MULTIPATH=y
CONFIG_CRYPTO_DEV_FSL_CAAM_TK_API=y

caam-keygen create randomkey ecb -s 16

ls -l /data/caam/
total 8
-rw-r--r-- 1 root root 36 apr 3 21.09 randomkey
-rw-r--r-- 1 root root 96 apr 3 21.09 randomkey.bb

cat /data/caam/randomkey | keyctl padd logon logkey: @s

dmsetup -v create $ENCRYPTED_LABEL --table "0 $(blockdev --getsz $DEVICE) crypt capi:tk(cbc(aes))-plain :36:logon:logkey: 0 $DEVICE 0 1 sector_size:512"

mkfs.ext4 -L $VOLUME_LABEL /dev/mapper/$ENCRYPTED_LABEL

mount /dev/mapper/$ENCRYPTED_LABEL ${MOUNT_POINT}

echo "Encrypted data" > ${MOUNT_POINT}/encrypted-file

umount $MOUNT_POINT
dmsetup remove $ENCRYPTED_LABEL
keyctl clear @s
reboot

caam-keygen import $KEYPATH/$KEYNAME.bb $IMPORTKEY
cat $IMPORTKEYPATH/$IMPORTKEY | keyctl padd logon logkey: @s
dmsetup -v create $ENCRYPTED_LABEL --table "0 $(blockdev --getsz $DEVICE) crypt capi:tk(cbc(aes))-plain :36:logon:logkey: 0 $DEVICE 0 1 sector_size:512"
mount /dev/mapper/$ENCRYPTED_LABEL ${MOUNT_POINT}

cat ${MOUNT_POINT}/encrypted-file
Encrypted data

Conclusion

Encryption could be hard, but the CAAM module makes it pretty much straight forward. It protect your secrets from physical attacks, which could be hard to protect otherwise.

However, keep in mind that as soon as the encrypted device is mounted and available to the system, it is free to read for any intruder that have access to the system.

The device security chain is no stronger than its weakest link and you have to identify and handle all potential security risks. This is only one.

References

Background

I do have a system where I can swap between iMX8M Mini and iMX8M Plus CPU modules on the same carrier board.

I did write a a SPI driver for a device on the carrier board. The device is connected to the ECSPI1 (the CPU contains several ECSPI modules) and use the hardware chipselect 0 (SS0). The driver has been used with the iMX8MM CPU module for a while, but as soon I swapped to the iMX8MP it certainly stopped working.

Both iMX8MM and iMX8MP have the same ECSPI IP block that is managed by the spi-imx [1] Linux kernel driver, the application and root filesystem is the same as well.

Same driver, same application, different module. What is happening?

The driver layer also did not report anything suspicious, all SPI transactions contained the data I expected and was successfully sent out on the bus. After debugging the application, driver and devicetree for a while, I took a closer look on the actual SPI signals.

SPI signals

I'm not going to describe the SPI interface specifications, please see Wikipedia [2] or such for more details.

It turns out that the chip select goes inactive after each sent byte, which is a weird behavior. The chipselect should stay low during the whole data transaction.

Here is the signals of one transaction of two bytes:

The ECSPI modules supports dynamic burst size, so I was experimenting with that without any success.

Workaround

The best workaround I came up with was to MUX the chipselect pin to the GPIO function instead of SS0 and map that GPIO as chipselect to ECSPI1 by override the affected properties in the device tree file:

&ecspi1 {
          cs-gpios =
                      <&gpio5 9 GPIO_ACTIVE_LOW>,
                      <&gpio2 8 GPIO_ACTIVE_LOW>;
};

&pinctrl_ecspi1_cs0 {
        fsl,pins = <
                MX8MP_IOMUXC_ECSPI1_SS0__GPIO5_IO09         0x40000
                    >;
};

Then the signals looks better:

Conclusion

I do not know if all ECSPI modules with all HW chipselects is affected or only SS0 @ ECSPI1. I could not find anything about it in the iMX8MP Errata.

The fact that the workaround did work makes me suspect a hardware bug in the iMX8MP processor. I guess we will see if it shows up in the errata later on.

References