TIL - Docker scratch image

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

The scratch image is the smallest possible image for docker. It does not contain any libraries nor other executables. It is simply a new, fresh and empty setup of namespaces.

The FROM scratch line is even a no-op [1] in the Dockerfile, which results in that it will not create an extra layer in you image. Example of a Dockerfile:

FROM scratch
ADD hello /
CMD ["/hello"]

As the Docker image does not contains any libraries, the hello application in the example above has to be compiled statically.

One use I see is to setup a Docker image based on a completely custom made root filesystem, e.g. the output from Buildroot.

See Docker documentation [2] for further reading.

TIL - Sort in VIM

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

After 15+ years of daily VIM usage I just discovered the :sort function. Quite embarassing.

It is also possible to sort in revese ( :sort!) and remove duplicate lines ( :sort u).

TIL - notmuch-lore

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

Notmuch [1] is an mail indexer and is a great tool to search in high-volume mailboxes (e.g. mailing lists). Being subscribed to all these mailing lists and retreiving all mails over IMAP daily could be quite annoying and harm your mail quota.

Therefor, public-inbox [2] makes archives of many of the public maling lists out there and makes it available in a git repository.

That is good, but you want to read your emails in your email-client, right?

TIL - git man-pages

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

I'm a big user of man-pages.

Today I found a set of pages that I've not noticed before when I was reading man git:

SEE ALSO
        gittutorial(7), gittutorial-2(7), giteveryday(7), gitcvs-migration(7), gitglossary(7), gitcore-tutorial(7),
       gitcli(7), The Git User’s Manual[1], gitworkflows(7)

Especially

• man gitcore-tutorial
• man giteveryday
• man gitglossary
• man gittutorial
• man gitworkflows

They are worth a look.

TIL - Git --color-moved

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

Did you know that Git is able to detect moved blocks and use different colors from the usual added/removed lines? Me neither until now.

The paremeter is --color-moved and has been around since v0.4.0, so there is no new feature.

This is from the git manual page [1] :

--color-moved[=<mode>]
Moved lines of code are colored differently. The <mode> defaults to no if the option is not given and to zebra if the option with no mode is given. The mode must be one of:

no
Moved lines are not highlighted.

default
Is a synonym for zebra. This may change to a more sensible mode in the future.

plain
Any line that is added in one location and was removed in another location will be colored with color.diff.newMoved. Similarly color.diff.oldMoved will be used for removed lines that are added somewhere else in the diff. This mode picks up any moved line, but it is not very useful in a review to determine if a block of code was moved without permutation.

blocks
Blocks of moved text of at least 20 alphanumeric characters are detected greedily. The detected blocks are painted using either the color.diff.{old,new}Moved color. Adjacent blocks cannot be told apart.

zebra
Blocks of moved text are detected as in blocks mode. The blocks are painted using either the color.diff.{old,new}Moved color or color.diff.{old,new}MovedAlternative. The change between the two colors indicates that a new block was detected.

dimmed-zebra
Similar to zebra, but additional dimming of uninteresting parts of moved code is performed. The bordering lines of two adjacent blocks are considered interesting, the rest is uninteresting. dimmed_zebra is a deprecated synonym.

The --color-moved argument can be used for any commands that generates a diff, such as git diff, git log -p, git stash show and more.

Example

This is an example of git diff where I have moved a function and made som small changes:

That the block is moved is clear, but it is not obvious what have changed in the block itself.

If we instead use --color-moved, we get a different style on the output:

The magenta and cyan is the moved block, and in addition, we get the red/green colors for removed/added lines.

$git config --global diff.colormoved default

TIL - Buildroot and LIBFOO_LINUX_CONFIG_FIXUPS

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

Some applications in a Linux system depends on certain kernel features to work properly. I'm currently working on adding support for CRIU [1] in Buildroot [2] which has such requirements.

That's when I stumble upon the LIBFOO_LINUX_CONFIG_FIXUP variable.

* +LIBFOO_LINUX_CONFIG_FIXUPS+ lists the Linux kernel configuration
  options that are needed to build and use this package, and without
  which the package is fundamentally broken. This shall be a set of
  calls to one of the kconfig tweaking option: `KCONFIG_ENABLE_OPT`,
  `KCONFIG_DISABLE_OPT`, or `KCONFIG_SET_OPT`.
  This is seldom used, as package usually have no strict requirements on
  the kernel options.

define CRIU_LINUX_CONFIG_FIXUPS
         $(call KCONFIG_ENABLE_OPT,CONFIG_CHECKPOINT_RESTORE)
endef

TIL - ignore_routes_with_linkdown

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

Usually, a high level network managers such as NetworkManager or Connman, removes routes for unaccessible ("linkdown" or "unplugged") links states.

In some cases, these links are not managed eiter by a network manager but are managed in other ways or in somehow misconfigured, then you can end up with a default route still hangs around even if the link is down.

It doesn't sound too bad, if it weren't for the fact that it is still active and if its metric value is lower than some other default route, it will be the chosen route.

Consider the following routes:

# ip route
default via 10.42.0.1 dev rndis proto static metric 100 linkdown
default via 192.168.30.1 dev br0 proto static metric 425
10.42.0.0/24 dev rndis proto kernel scope link src 10.42.0.20 metric 100 linkdown
192.168.30.0/23 dev br0 proto static scope link metric 425
192.168.30.0/23 dev br0 proto kernel scope link src 192.168.30.97 metric 425

Even if the rndis interface has the "linkdown" flag, Linux will use it as default route regardless (lowest metric value) for any FIB (Forward Information Base) lookup.

The kernel provide a feature to ignore routes that use links with the down state; ignore_routes_with_linkdown.

This is a rather old feature [1] introduced in v4.2 and documented [3] as follows:

ignore_routes_with_linkdown - BOOLEAN
        Ignore routes whose link is down when performing a FIB lookup.

When set, the route will not only be set as linkdown but dead linkdown to indicate that the route is currently ignored during a FIB lookup.

The flag could be set with some granuality, both for IPv4/IPv6, all interfaces or just on a single interface. For example:

$sysctl -a | grep ignore_routes_with_link
net.ipv4.conf.all.ignore_routes_with_linkdown = 0
net.ipv4.conf.default.ignore_routes_with_linkdown = 0
net.ipv4.conf.docker0.ignore_routes_with_linkdown = 0
net.ipv4.conf.enp0s13f0u2u4.ignore_routes_with_linkdown = 0
net.ipv4.conf.enp0s31f6.ignore_routes_with_linkdown = 0
net.ipv4.conf.eth0.ignore_routes_with_linkdown = 0
net.ipv4.conf.lo.ignore_routes_with_linkdown = 0
net.ipv4.conf.tun0.ignore_routes_with_linkdown = 0
net.ipv4.conf.wlp0s20f3.ignore_routes_with_linkdown = 0
net.ipv6.conf.all.ignore_routes_with_linkdown = 0
net.ipv6.conf.default.ignore_routes_with_linkdown = 0
net.ipv6.conf.docker0.ignore_routes_with_linkdown = 0
net.ipv6.conf.enp0s13f0u2u4.ignore_routes_with_linkdown = 0
net.ipv6.conf.enp0s31f6.ignore_routes_with_linkdown = 0
net.ipv6.conf.eth0.ignore_routes_with_linkdown = 0
net.ipv6.conf.lo.ignore_routes_with_linkdown = 0
net.ipv6.conf.tun0.ignore_routes_with_linkdown = 0
net.ipv6.conf.wlp0s20f3.ignore_routes_with_linkdown = 0

TIL - Split streams with boost::tee_device

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

Split streams could be useful if you want the same output to appear in more than one stream at once. Boost support tee_device, which works pretty much as the tee(1) [1] command line tool. Everything written to that device is splitted up and written into two streams. Think of the letter T.

Here is an example of how to let everything written to cout and also be logged into a file. This is achieved by change the streambuffer for cout to instead use the logger stream:

    #include <fstream>
    #include <iostream>

    #include <boost/iostreams/tee.hpp>
    #include <boost/iostreams/stream.hpp>

    using std::ostream;
    using std::ofstream;
    using std::cout;
    using std::endl;

    using boost::iostreams::tee_device;
    using boost::iostreams::stream;

    // Create typedefs to avoid too long type declarations later on
    typedef tee_device<ostream, ofstream> TeeDevice;
    typedef stream<TeeDevice> TeeStream;

    int main()
    {
        // Create streams and connect them to a file and stdout
        ofstream ofs("stdout.log");
        ostream tmp(cout.rdbuf());

        TeeDevice logdevice(tmp, ofs);
        TeeStream logger(logdevice);

        // Set streambuffer for cout 
        cout.rdbuf(logger.rdbuf());

        // Make some logs
        cout << "Log output to both stdout and file." << endl;
    }

    [08:35:24]marcus@goliat:~/tmp/tee$ g++ main.cpp -o main && ./main
    Log output to both stdout and file.

    [08:56:09]marcus@goliat:~/tmp/tee$ cat stdout.log 
    Log output to both stdout and file.

TIL - NFS UDP Support

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

Today I was trying to bring up an really old board (v2.6.32 kernel) that used to mount its root filesystem via NFS. Unfortunately, I have to stick to this configuration for a few reasons.

The virtual machine that came with the board did not start anymore, so I had to setup a new one.

Playing with such old stuff is not fun at all. Really. You struggle with such energy draining tasks like find old toolchains, add support for diffie-hellman-group1-sha1 to get SSH to work, and wierd NFS configurations to make it compatible.

Anyway.

One thing I was struggle with for a while was this error:

Root-NFS: Unable to get nfsd port number from server, using default

It turned out that that the kernel tries to communicate with the NFS server via UDP, which is disabled by default since v5.5 by the CONFIG_NFS_DISABLE_UDP_SUPPORT configuration option:

config NFS_DISABLE_UDP_SUPPORT
       bool "NFS: Disable NFS UDP protocol support"
       depends on NFS_FS
       default y
       help
     Choose Y here to disable the use of NFS over UDP. NFS over UDP
     on modern networks (1Gb+) can lead to data corruption caused by
     fragmentation during high loads.

As it is set by default, I had to find a distribution that used a kernel older than v5.5 for my virtual machine, unless I do not want to compile the kernel myself - which I don't. You can tell the kernel to mount NFS via TCP by kernel command line options, but I was not allowed to change it.

This is not fun.

TIL - Buildroot & BR_NO_CHECK_HASH_FOR

TIL, Today I Learned, is more of a "I just figured this out: here are my notes, you may find them useful too" rather than a full blog post

In Buildroot [1], the integrity of (allmost) all downloaded packages is verified against a hash. Even packages that are fetched from a git repository is verified this way.

This is a good thing that no one really should work around.

Today I had a debug-session for one package which I locally cloned and frequently made changes to. I told Buildroot to fetch the source code for the package locally to speed up my iterations between new code changes and testing on target.

The hash did of course change after each code change, which became quite annoying.

[22:58:16]marcus@goliat:~/git/buildroot$ git grep BR_NO_CHECK_HASH_FOR docs/
[22:58:18]marcus@goliat:~/git/buildroot$

ifeq ($(BR2_LINUX_KERNEL)$(BR2_LINUX_KERNEL_LATEST_VERSION),y)
BR_NO_CHECK_HASH_FOR += $(LINUX_SOURCE)
endif

LINUX_PATCHES = $(call qstrip,$(BR2_LINUX_KERNEL_PATCH))

# We have no way to know the hashes for user-supplied patches.
BR_NO_CHECK_HASH_FOR += $(notdir $(LINUX_PATCHES))