TIL - Split streams with boost::tee_device

TIL - Split streams with boost::tee_device

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

Split streams could be useful if you want the same output to appear in more than one stream at once. Boost support tee_device, which works pretty much as the tee(1) [1] command line tool. Everything written to that device is splitted up and written into two streams. Think of the letter T.

/media/boost-teedevice.png

Here is an example of how to let everything written to cout and also be logged into a file. This is achieved by change the streambuffer for cout to instead use the logger stream:

    #include <fstream>
    #include <iostream>

    #include <boost/iostreams/tee.hpp>
    #include <boost/iostreams/stream.hpp>

    using std::ostream;
    using std::ofstream;
    using std::cout;
    using std::endl;

    using boost::iostreams::tee_device;
    using boost::iostreams::stream;

    // Create typedefs to avoid too long type declarations later on
    typedef tee_device<ostream, ofstream> TeeDevice;
    typedef stream<TeeDevice> TeeStream;

    int main()
    {
        // Create streams and connect them to a file and stdout
        ofstream ofs("stdout.log");
        ostream tmp(cout.rdbuf());

        TeeDevice logdevice(tmp, ofs);
        TeeStream logger(logdevice);

        // Set streambuffer for cout 
        cout.rdbuf(logger.rdbuf());

        // Make some logs
        cout << "Log output to both stdout and file." << endl;
    }

Compile and test

    [08:35:24]marcus@goliat:~/tmp/tee$ g++ main.cpp -o main && ./main
    Log output to both stdout and file.

    [08:56:09]marcus@goliat:~/tmp/tee$ cat stdout.log 
    Log output to both stdout and file.

TIL - NFS UDP Support

TIL - NFS UDP Support

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

Today I was trying to bring up an really old board (v2.6.32 kernel) that used to mount its root filesystem via NFS. Unfortunately, I have to stick to this configuration for a few reasons.

The virtual machine that came with the board did not start anymore, so I had to setup a new one.

Playing with such old stuff is not fun at all. Really. You struggle with such energy draining tasks like find old toolchains, add support for diffie-hellman-group1-sha1 to get SSH to work, and wierd NFS configurations to make it compatible.

Anyway.

One thing I was struggle with for a while was this error:

Root-NFS: Unable to get nfsd port number from server, using default

It turned out that that the kernel tries to communicate with the NFS server via UDP, which is disabled by default since v5.5 by the CONFIG_NFS_DISABLE_UDP_SUPPORT configuration option:

config NFS_DISABLE_UDP_SUPPORT
       bool "NFS: Disable NFS UDP protocol support"
       depends on NFS_FS
       default y
       help
     Choose Y here to disable the use of NFS over UDP. NFS over UDP
     on modern networks (1Gb+) can lead to data corruption caused by
     fragmentation during high loads.

As it is set by default, I had to find a distribution that used a kernel older than v5.5 for my virtual machine, unless I do not want to compile the kernel myself - which I don't. You can tell the kernel to mount NFS via TCP by kernel command line options, but I was not allowed to change it.

This is not fun.

/media/old-waste.png

TIL - Buildroot & BR_NO_CHECK_HASH_FOR

TIL - Buildroot & BR_NO_CHECK_HASH_FOR

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

In Buildroot [1], the integrity of (allmost) all downloaded packages is verified against a hash. Even packages that are fetched from a git repository is verified this way.

This is a good thing that no one really should work around.

Today I had a debug-tracking-session for one package which I locally cloned and frequently made changes to. I told Buildroot to fetch the source code for the package locally to speed up my iterations between new code changes and testing on target.

The hash did of course change after each code change, which became quite annoying.

BR_NO_CHECK_HASH_FOR

The BR_NO_CHECK_HASH_FOR was something I found by a coincidence when looking into the support/download/check-hash file.

No wonder why I have missed this one, it is not mention in the documentation at all:

[22:58:16]marcus@goliat:~/git/buildroot$ git grep BR_NO_CHECK_HASH_FOR docs/
[22:58:18]marcus@goliat:~/git/buildroot$

It does what it says - do not check hash for a certain list of files.

Here is an example on how the linux package make use of BR_NO_CHECK_HASH_FOR:

ifeq ($(BR2_LINUX_KERNEL)$(BR2_LINUX_KERNEL_LATEST_VERSION),y)
BR_NO_CHECK_HASH_FOR += $(LINUX_SOURCE)
endif

LINUX_PATCHES = $(call qstrip,$(BR2_LINUX_KERNEL_PATCH))

# We have no way to know the hashes for user-supplied patches.
BR_NO_CHECK_HASH_FOR += $(notdir $(LINUX_PATCHES))

Conclusion

I found this useful for my bug-hunting, but as a general rule, this is something that you probably should not use.

TIL - Git jump

TIL - Git jump

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

The Git v2.40.0 was released [1] yesterday (2023-03-13) and one of the release notes that caught my eyes was this one:

* "git jump" (in contrib/) learned to present the "quickfix list" to
  its standard output (instead of letting it consumed by the editor
  it invokes), and learned to also drive emacs/emacsclient.

I'm familiar with the "quickfix list" in Vim as I have been using it for many years, I even wrote a small post [2] about it looong time ago.

But what about git jump thing?

What is "quickfix list"

First a few words about quickfix. Quickfix is a list of "interesting" locations in a project such as compilation errors, lint warnings or whatever you want to feed the list with.

Vim has several commands for navigating the quickfix list, here is the most common ones:

  • :copen - Open the quickfix list window.
  • :ccl or :cclose - Close the quickfix list window.
  • :cnext or :cn - Go to the next item on the list.
  • :cprev or :cp - Go to the previous item on the list.
  • :cfirst - Go to the first item on the list.
  • :clast - Go to the last item on the list.

I almost exclusively use the :copen in my daily work. The quickfix list window let you iterate through all items and easily navigate to selected position.

Example on a quickfix window generated with git jump grep compatible:

/media/vim-quickfix.jpg

Git-jump

Git-jump [3] is a contribution script (has it's location in the ./contrib directory of the git repository) and has actually been around since 2011 without me noticing.

Git-jump can help you generate "interesting" quickfix lists and feed it to vim directly. It currently support these lists:

  • The beginning of any diff hunks.

    git jump diff
    
  • The beginning of any merge conflict markers.

    git jump merge
    
  • Any grep matches, including the column of the first match on a line.

    git jump grep compatible
    
  • Any whitespace errors detected by git diff --check.

The script does also supports Emacs, for those who like that kind of thing.

Conclusion

I have only been aware of this tool for a day and it is already totally integrated in my daily workflow.

The commands that I use most are git jump diff and git jump grep. Those are just awesome.

TIL - U-Boot support for HTTP

TIL - U-Boot support for HTTP

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

Until now, U-Boot did only support UDP and were limited to protocols that are based on that. In practice it meant that you were only able to use TFTP and NFS for network file transfer.

Ying-Chun Liu has recently added support for HTTP [1] and the wget [2] command which opens up for a range of new protocols.

Setup

I do currenty test TCP/HTTP by using it on my i.MX8 eval kit and it works pretty well.

If you just want to test the functionality, you can do it virtually with qemu. It requires that you have both qemu and a cross toolchain for aarch64 installed on your host system.

Clone the repository:

https://github.com/u-boot/u-boot.git

Use qemu_arm64_defconfig as base:

make qemu_arm64_defconfig

Enable the following configurations in order to support TCP and HTTP:

  • CONFIG_CMD_WGET=y
  • CONFIG_PROT_TCP=y

Build U-Boot:

make CROSS_COMPILE=aarch64-linux-gnu-

Start qemu:

qemu-system-aarch64 -machine virt -cpu cortex-a57 -bios u-boot.bin

Assuming that you already have a HTTP server up and running on your host, you can now fetch the image with wget:

wget $loadaddr 192.168.1.84:/uImage

Or by using serverip:

setenv serverip 192.168.1.84
wget $loadaddr /uImage
/media/u-boot-http.png

Conclusion

This is a great step in the right direction!

Only HTTP is supported for now, which makes it unsuitable for any public website, but HTTPS integreated with UEFI will follows and then we have a standardised way to download boot images!

TIL - Virtual CAN interfaces

TIL - Virtual CAN interfaces

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

There are many scenarios where a virtual Controller Area Network(CAN) interface could be handy. It let you develop and test applications that without an actual physical CAN bus.

My scenario is to play around with Berkley Packet Filter(BPF) [1] and do some data manipulation on recieved CAN frames in kernel space.

Setup

We use iproute2 [2] in order to create the virtual CAN interfaces

sudo ip link add dev vcan0 type vcan
sudo ip link add dev vcan1 type vcan

Bring up the interaces

sudo ip link set up vcan0
sudo ip link set up vcan1

You are now able to use can-utils [3] to send and recieve CAN frames.

Routing

If you have enabled the CONFIG_CAN_GW in your kernel, then you can create gateway rules and route frames between CAN interfaces cangw (part of can-utils).

Verify that CAN_GW is supported by the kernel

[16:25:10]marcus@goliat:~/git/linux$ zgrep CONFIG_CAN_GW /proc/config.gz
CONFIG_CAN_GW=m

Load kernel module

sudo modprobe can-gw

Create gateway rule to route messages from vcan0 to vcan1 and vice versa

sudo cangw -A -s vcan0 -d vcan1 -e
sudo cangw -A -s vcan1 -d vcan0 -e

The routing is not limited to only virtual CAN interfaces. You can setup gateway rules between real and virtual ones.

TIL - Debug FindBoost in CMake

TIL - Debug FindBoost in CMake

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

When you use find_package for a Boost component in CMake it will end up with the FindBoost.cmake [1] module. It is usually located in /usr/share/cmake/Modules/FindBoost.cmake.

I had to debug a cross compile setup for an application using CMake, and the root file system did use wierd paths for everything.

There are a few variables you could set (all of those are mention in the FindBoost.cmake file) to make your debugging easier.

The first one is Boost_DEBUG, it is described as follows

``Boost_DEBUG``
  Set to ``ON`` to enable debug output from ``FindBoost``.
  Please enable this before filing any bug report.

It will give you verbose information during the build process and print all the search paths it goes through.

Once you find out what is not right with your setup, you can hand over some hints where the CMake buildsystem should search

``BOOST_ROOT``, ``BOOSTROOT``
  Preferred installation prefix.

``BOOST_INCLUDEDIR``
  Preferred include directory e.g. ``<prefix>/include``.

``BOOST_LIBRARYDIR``
  Preferred library directory e.g. ``<prefix>/lib``.

``Boost_NO_SYSTEM_PATHS``
  Set to ``ON`` to disable searching in locations not
  specified by these hint variables. Default is ``OFF``.

``Boost_ADDITIONAL_VERSIONS``
  List of Boost versions not known to this module.
  (Boost install locations may contain the version).

TIL -Yocto bitbake-getvar

TIL - Yocto bitbake-getvar

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

Writing recipes for Yocto has recently become something I do several times a week. All tools that help you with debugging is allways appreciated, and one such tool I found recently is bitbake-getvar that came with the Honister release of Yocto.

It is part of the bitbake suite and will give you variable values for a certain recipe, both unexpanded and expanded.

For example

builder@b4a399a9615f:/build$ bitbake-getvar -r imx-atf EXTRA_OEMAKE
#
# $EXTRA_OEMAKE [6 operations]
#   set /work/layers/poky/meta/conf/bitbake.conf:553
#     ""
#   :prepend[task-compile] /work/layers/poky/meta/conf/bitbake.conf:606
#     "${PARALLEL_MAKE} "
#   :prepend[task-install] /work/layers/poky/meta/conf/bitbake.conf:610
#     "${PARALLEL_MAKEINST} "
#   set /work/layers/poky/meta/conf/documentation.conf:167
#     [doc] "Additional GNU make options."
#   append /work/layers/meta-freescale/recipes-bsp/imx-atf/imx-atf_2.4.bb:26
#     "     CROSS_COMPILE="${TARGET_PREFIX}"     PLAT=${ATF_PLATFORM} "
#   :append[sm2s-imx8mm] /work/layers/meta-msc/recipes-bsp/imx-atf/imx-atf_2.4.bbappend:9
#     "         IMX_BOOT_UART_BASE="${IMX_BOOT_UART_BASE}"      "
# pre-expansion value:
#   "      CROSS_COMPILE="${TARGET_PREFIX}"     PLAT=${ATF_PLATFORM}    IMX_BOOT_UART_BASE="${IMX_BOOT_UART_BASE}"      "
EXTRA_OEMAKE="      CROSS_COMPILE=\"aarch64-imx8evk-linux-\"     PLAT=imx8mm         IMX_BOOT_UART_BASE=\"0x30860000\"       "

Handy and useful.

TIL - systemd and $MAINPID

TIL - SystemD and $MAINPID

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

A special environmnent variable, $MAINPID, is set and is available for the .service file. The variable exposes the MainPID property of the service and could be useful if your custom services e.g. forks.

This let you do

ExecReload=/bin/kill -HUP "$MAINPID"

To send a HUP signal to the main process created by your service.

You can read out the MainPID property of a process by using systemctl:

systemctl [--user] show --property MainPID --value <service>

Read more about it in the systemd.service manual [1].