IEEE 802.11

We will see 802.11 a lot of times, so the first thing is to know where these numbers comes from. IEEE 802.11 is a set of specifications for implementation of wireless networking over several frequency bands. The specifications cover layer 1 (Physical) and layer 2 (Data link) of the OSI model [3].

The Linux kernel MAC subsystem register ieee80211 compliant hardware device with

int ieee80211_register_hw(struct ieee80211_hw *hw)

found in .../net/mac80211/main.c

The Management Layer (MLME)

One more thing that we need to cover is the management layer, since all other layers somehow depend on it.

There are three components in the 802.11 management architecture: - The Physical Layer Management Entity (PLME) - The System Management Entity (SME) - The MAC Layer Management Entity (MLME)

The Management layer assist you in several ways. For instance, it handle things such as scanning, authentication, beacons, associations and much more.

void ieee80211_send_probe_req(struct ieee80211_sub_if_data *sdata,
                  const u8 *src, const u8 *dst,
                  const u8 *ssid, size_t ssid_len,
                  const u8 *ie, size_t ie_len,
                  u32 ratemask, bool directed, u32 tx_flags,
                  struct ieee80211_channel *channel, bool scan)

void ieee80211_send_auth(struct ieee80211_sub_if_data *sdata,
             u16 transaction, u16 auth_alg, u16 status,
             const u8 *extra, size_t extra_len, const u8 *da,
             const u8 *bssid, const u8 *key, u8 key_len, u8 key_idx,
             u32 tx_flags)

static void ieee80211_send_assoc(struct ieee80211_sub_if_data *sdata)

MAC (Medium Access Control)

All ieee80211 devices needs to implement the Management Layer (MLME), but the implementation could be in device hardware or software. These types of devices are divided into Full MAC device (hardware implementation) and Soft MAC device (software implementation). Most devices today are soft MAC devices.

The MAC layer can be further broken down into two pieces: Upper MAC and Lower MAC. The upper part of the MAC handle the management aspect (all that we covered in the MLME section above), and the lower part handle the time critical operations such as ACK:ing received packets.

Linux does only handle the upper part of MAC, the lower part is operated in device hardware. What we can see in the figure is that the MAC layer is separating data packets from configuration/management packets. The data packets is forwarded to the network device and will travel the same path through the network layer as data packets from all other type of network devices.

The Linux wireless subsystem consists of two major parts, where this, mac80211, is one of them. cfg80211 is the other major part.

CFG80211

cfg80211 is a configuration management service for mac80211 compliant devices. Both Full MAC and Soft MAC devices needs to implement operations to be compatible with the cfg80211 configuration interface in order to let userspace application to configure the device.

The configuration may be done with on of two interfaces, wext and nl80211.

Conclusion

The whole network stack of the Linux kernel is really complex and optimized for high throughput with low latencies. In this post we only covered what support for wireless devices has complemented the stack with, which is mainly the mac80211 layer for handle all device management, and cfg80211 layer to configure the MAC layer. Packets to wireless devices is divided into data packets and configuration/managment packets. The data packets follow the same path as for all network devices, and the management packets goes to the cfg80211 layer.

References

Reverse engineering the adapter

The reverse engineering was quite simple. It turns out that the adapter only has 1 configuration, 1 interface and 1 endpoint of in-interrupt type. This simply means that it only has an unidirectional communication path, initiated by sending an interrupt URB (USB Request Block) to the device. If you are not familiar with what configurations, interfaces and endpoints are in terms of USB, please google the USB standard specification.

The data from our URB was only 8 bytes. After some testing with my RC controller I got the following mapping between data and the channels on the controller:

data[0] = channel 1
data[1] = ? (Possibly a switch)
data[2] = channel 2
data[3] = channel 3
data[4] = channel 4
data[5] = channel 5
data[6] = channel 6
data[7] = channel 7

So I created a device driver that registered an input device with the following events:

Using a simulator

Heli-X [2] is an excellent cross platform flight simulator that runs perfect on Linux. Now I have spent several hours with a Goblin 700 Helicopter and it is just as fun as I rembembered.

Available in Linux kernel 4.17

Of course all code is submitted to the Linux kernel and should be merged in v4.17.

References

References

The old implementation

Before v2.6.36, the calculation of badness score tried to be smarter, besides looking for the total memory usage (task->mm->total_vm), it also considered: - Whether the process creates a lot of children - Whether the process has been running for a long time, or has used a lot of CPU time - Whether the process has a low nice value - Whether the process is privileged (CAP_SYS_ADMIN or CAP_SYS_RESOURCE set) - Whether the process is making direct hardware access

At first glance, all these criteria looks valid, but if you think about it a bit, there is a lot of pitfalls here which makes the selection not so fair. For example: A process that creates a lot of children and consumes some memory could be a leaky webserver. Another process that fits into the description is your session manager for your desktop environment which naturally creates a lot of child processes.

The new implementation

This heuristic selection has evolved over time, instead of looking on mm->total_vm for each task, the task's RSS (resident set size, [2]) and swap space is used instead. RSS and Swap space gives a better indication of the amount that we will be able to free if we chose this task. The drawback with using mm->total_vm is that it includes overcommitted memory ( see [3] for more information ) which is pages that the process has claimed but has not been physically allocated.

The process is now only counted as privileged if CAP_SYS_ADMIN is set, not CAP_SYS_RESOURCE as before.

The code

The whole implementation of OOM killer is located in mm/oom_kill.c. The function oom_badness() will be called for each task in the system and returns the calculated badness score.

Let's go through the function.

unsigned long oom_badness(struct task_struct *p, struct mem_cgroup *memcg,
              const nodemask_t *nodemask, unsigned long totalpages)
{
    long points;
    long adj;

    if (oom_unkillable_task(p, memcg, nodemask))
        return 0;

Looking for unkillable tasks such as the global init process.

p = find_lock_task_mm(p);
if (!p)
    return 0;

adj = (long)p->signal->oom_score_adj;
if (adj == OOM_SCORE_ADJ_MIN ||
        test_bit(MMF_OOM_SKIP, &p->mm->flags) ||
        in_vfork(p)) {
    task_unlock(p);
    return 0;
}

If proc/[pid]/oom_score_adj is set to OOM_SCORE_ADJ_MIN (-1000), do not even consider this task

points = get_mm_rss(p->mm) + get_mm_counter(p->mm, MM_SWAPENTS) +
    atomic_long_read(&p->mm->nr_ptes) + mm_nr_pmds(p->mm);
task_unlock(p);

Calculate a score based on RSS, pagetables and used swap space

if (has_capability_noaudit(p, CAP_SYS_ADMIN))
    points -= (points * 3) / 100;

If it is root process, give it a 3% discount. We are no mean people after all

adj *= totalpages / 1000;
points += adj;

Normalize and add the oom_score_adj value

return points > 0 ? points : 1;

At last, never return 0 for an eligible task as it is reserved for non killable tasks

}

Conclusion

The OOM logic is quite straightforward and seems to have been stable for a long time (v2.6.36 was released in october 2010). The reason why I was looking at the code was that I did not think the behavior I saw when experimenting corresponds to what was written in the man page for oom_score. It turned out that the manpage was not updated when the new calculation was introduced back in 2010.

I have updated the manpage and it is available in v4.14 of the Linux manpage project [4].

commit 5753354a3af20c8b361ec3d53caf68f7217edf48
Author: Marcus Folkesson <marcus.folkesson@gmail.com>
Date:   Fri Nov 17 13:09:44 2017 +0100

    proc.5: Update description of /proc/<pid>/oom_score

    After Linux 2.6.36, the heuristic calculation of oom_score
    has changed to only consider used memory and CAP_SYS_ADMIN.

    See kernel commit a63d83f427fbce97a6cea0db2e64b0eb8435cd10.

    Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>
    Signed-off-by: Michael Kerrisk <mtk.manpages@gmail.com>

diff --git a/man5/proc.5 b/man5/proc.5
index 82d4a0646..4e44b8fba 100644
--- a/man5/proc.5
+++ b/man5/proc.5
@@ -1395,7 +1395,9 @@ Since Linux 2.6.36, use of this file is deprecated in favor of
 .IR /proc/[pid]/oom_score_adj .
 .TP
 .IR /proc/[pid]/oom_score " (since Linux 2.6.11)"
-.\" See mm/oom_kill.c::badness() in the 2.6.25 sources
+.\" See mm/oom_kill.c::badness() in pre 2.6.36 sources
+.\" See mm/oom_kill.c::oom_badness() after 2.6.36
+.\" commit a63d83f427fbce97a6cea0db2e64b0eb8435cd10
 This file displays the current score that the kernel gives to
 this process for the purpose of selecting a process
 for the OOM-killer.
@@ -1403,7 +1405,16 @@ A higher score means that the process is more likely to be
 selected by the OOM-killer.
 The basis for this score is the amount of memory used by the process,
 with increases (+) or decreases (\-) for factors including:
-.\" See mm/oom_kill.c::badness() in the 2.6.25 sources
+.\" See mm/oom_kill.c::badness() in pre 2.6.36 sources
+.\" See mm/oom_kill.c::oom_badness() after 2.6.36
+.\" commit a63d83f427fbce97a6cea0db2e64b0eb8435cd10
+.RS
+.IP * 2
+whether the process is privileged (\-);
+.\" More precisely, if it has CAP_SYS_ADMIN or (pre 2.6.36) CAP_SYS_RESOURCE
+.RE
+.IP
+Before kernel 2.6.36 the following factors were also used in the calculation of oom_score:
 .RS
 .IP * 2
 whether the process creates a lot of children using
@@ -1413,10 +1424,7 @@ whether the process creates a lot of children using
 whether the process has been running a long time,
 or has used a lot of CPU time (\-);
 .IP *
-whether the process has a low nice value (i.e., > 0) (+);
-.IP *
-whether the process is privileged (\-); and
-.\" More precisely, if it has CAP_SYS_ADMIN or CAP_SYS_RESOURCE
+whether the process has a low nice value (i.e., > 0) (+); and
 .IP *
 whether the process is making direct hardware access (\-).
 .\" More precisely, if it has CAP_SYS_RAWIO

References

How it works

The code

asmlinkage __visible int printk(const char *fmt, ...)
{
    va_list args;
    int r;

    va_start(args, fmt);
    r = vprintk_func(fmt, args);
    va_end(args);

    return r;
}

__printf(1, 0) int vprintk_func(const char *fmt, va_list args)
{
    if (this_cpu_read(printk_context) & PRINTK_NMI_CONTEXT_MASK)
        return vprintk_nmi(fmt, args);

    if (this_cpu_read(printk_context) & PRINTK_SAFE_CONTEXT_MASK)
        return vprintk_safe(fmt, args);

    if (this_cpu_read(printk_context) & PRINTK_NMI_DEFERRED_CONTEXT_MASK)
        return vprintk_deferred(fmt, args);

    return vprintk_default(fmt, args);
}

asmlinkage int vprintk_emit(int facility, int level,
                const char *dict, size_t dictlen,
                const char *fmt, va_list args)
{

    ...

    <<<<< Strip kernel syslog prefix >>>>>

    ...

    <<<<< log_output() does the actual printing to __log_buf >>>>>
    printed_len = log_output(facility, level, lflags, dict, dictlen, text, text_len);

    ...

    if (!in_sched) {
        /*
         * Try to acquire and then immediately release the console
         * semaphore.  The release will print out buffers and wake up
         * /dev/kmsg and syslog() users.
         */
        if (console_trylock())
            console_unlock();
    }

    return printed_len;
}

if (console_trylock())
    console_unlock();

void console_unlock(void)
{
    ...


    <<<<< Endless loop? >>>>>
    for (;;) {

        <<<<< Go through all new messages >>>>>

        ...

        <<<<< Print to all consoles >>>><
        call_console_drivers(ext_text, ext_len, text, len);

        ...
    }

    ...

    <<<<<  Release the exclusive_console once it is used >>>>>
    console_locked = 0;

    ...

    <<<<< Wake up klogd >>>>>
    if (wake_klogd)
        wake_up_klogd();
}

static void call_console_drivers(const char *ext_text, size_t ext_len,
                 const char *text, size_t len)
{
    struct console *con;

    trace_console_rcuidle(text, len);

    if (!console_drivers)
        return;

    for_each_console(con) {
        if (exclusive_console && con != exclusive_console)
            continue;
        if (!(con->flags & CON_ENABLED))
            continue;
        if (!con->write)
            continue;
        if (!cpu_online(smp_processor_id()) &&
            !(con->flags & CON_ANYTIME))
            continue;
        if (con->flags & CON_EXTENDED)
            con->write(con, ext_text, ext_len);
        else
            con->write(con, text, len);
    }
}

trace_printk()

As we see, there is a lot of logic involved in a simple call to printk() and you should not be surprised if all your printing has impact on your systems performance or timing. But how do we debug if printk() is a no-no? The answer is trace_printk().

This function write (almost) directly to a trace buffer and is therefore a fairly fast operation. The trace buffer is exposed from tracefs, usually mounted at /sys/kernel/tracing.

As a bonus, the messages is merged with other output from ftrace when doing a function trace.

Other things that is good to know about __log_buf

[09:59:31]marcus@little:~/git/linux$ grep __log_buf System.map
c14cfba8 b __log_buf

How does mmap(2) work?

Every page table entry has a bit that tells us if the entry is valid in supervisor mode (kernel mode) only. And sure, all memory allocated in kernel space will have this bit set. What the mmap(2) system call do is simply creating a new page table entry with a different virtual address that points to the same physical memory page. The difference is that this supervisor-bit is not set. This let userspace access the memory as if it was a part of the application, for now it is! The kernel is not involved in those accesses at all, so it is really fast.

Magic? Kind of. The magic is called remap_pfn_range(). What remap_pfn_range() do is just essentially to update the process's specific page table with these new entries.

Example, please

unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order)

Now some code

priv->a_size = ATTRIBUTE_N * ATTRIBUTE_SIZE;

/* page align */
priv->a_size = PAGE_ALIGN(priv->a_size);
priv->a_area =vmalloc(priv->a_size);

static int scan_mmap (struct file *file, struct vm_area_struct *vma)
{
struct mmap_priv *priv = file->private_data;
unsigned long start = vma->vm_start;
unsigned long offset = vma->vm_pgoff << PAGE_SHIFT;
unsigned long page;
size_t size = vma->vm_end - vma->vm_start;
if (size > priv->a_size)
           return -EINVAL;
page = vmalloc_to_pfn((void *)priv->a_area);
if (remap_pfn_range(vma, start, page, priv->a_size, PAGE_SHARED))
           return -EAGAIN;
vma->vm_flags |= VM_RESERVED; /* avoid to swap out this VMA */
return 0;
}

struct page

A page is the smallest unit that matters in terms of virtual memory. This is because the MMU (Memory Management Unit, described in a upcoming post) is only dealing with those pages. A typical size for a page is 4KB, at least for 32bit architectures. The most 64-bit architectures uses 8KB pages.

Every one of those physical pages is represented by a struct page that is defined in include/linux/mm_types.h. That is a lot of pages. If we do a simple calculation: We have a 32-bit system that has 512MB of physical memory, this memory is divided into 131,072 4KB pages. Think of that 512MB is not even so much memory on a modern system today.

What I want to say is that this struct page should be kept as small as possible because it scales up a lot when physical memory increases.

Ok, so there is a struct page somewhere that got allocated for each physical page, which is a lot, but what does it do? It does a lot of housekeeping, lets look at a few set of members that I think is most interresting:

struct page {
    unsigned long flags;
    unsigned long private;
    void    *virtual;
    atomic_t    _count;
    pgoff_t    index;
#if USE_SPLIT_PTE_PTLOCKS
#if ALLOC_SPLIT_PTLOCKS
    spinlock_t  *ptl;
#else
    spinlock_t  ptl;
#endif
#endif
};

flags is keeping track of the page status which could be dirty (need to be written to media), locked in memory (not allowed to be paged out), permissions and so on. See enum pageflags in include/linux/page-flags.h for more information.

private is not a defined field. May be used as a long or interpreted as a pointer. (Shared with ptl in a union!)

virtual is the virtual address of the page. In case that the page belongs to the high memory (memory that is not permanently mapped) this field will be NULL and require dynamic mapping.

_count is a simple reference counter to determine when the page is free for allocation.

index is the offset within a mapping.

ptl is a interresting one! I think it requires a special section in this post. (Shared with private in a union!)

#define USE_SPLIT_PTE_PTLOCKS (NR_CPUS >= CONFIG_SPLIT_PTLOCK_CPUS)
#define USE_SPLIT_PMD_PTLOCKS (USE_SPLIT_PTE_PTLOCKS && \
    IS_ENABLED(CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK))
#define ALLOC_SPLIT_PTLOCKS (SPINLOCK_SIZE > BITS_PER_LONG/8)

What is PID 1

The top-most process in a UNIX system has PID (Process ID) 1 and is usually the init process. The Init process is the first userspace application started on a system and is started by the kernel at boottime. The kernel is looking in a few predefined paths (and the init kernel parameter). If no such application is found, the system will panic().

See init/main.c:kernel_init

if (!try_to_run_init_process("/sbin/init") ||
    !try_to_run_init_process("/etc/init") ||
    !try_to_run_init_process("/bin/init") ||
    !try_to_run_init_process("/bin/sh"))
    return 0;
panic("No working init found.  Try passing init= option to kernel. "
      "See Linux Documentation/admin-guide/init.rst for guidance.");

All processes in UNIX has a parent/child relationship which builds up a big relationship-tree. Some resources and permissions are inherited from parent to child such as UID and cgroup restrictions.

As in the real world, with parenthood comes obligations. For example: What is usually the last line of your main()-function? Hopfully something like

return EXIT_SUCCESS;

All processes exits with an exit code that tells us if the operation was sucessful or not. Who is interested in this exit code anyway? In the real world, the parents are interested in their children's result, and so even here. The parent is responsible to wait(2) on their children to terminate just to fetch its exit code. But what if the parent died before the child?

Lets go back to the init process. The init process has several tasks, and one is to adopt "orphaned" (called zombie) child processes. Why? Because all processes will return an exit code and will not terminate completely until someone is listen for what they have to say. The init process is simply wait(2):ing on the exit code, throw it away and let the child die. Sad but true, but the child may not rest i peace otherwise. The operating system expects the init process to reap adopted children. Otherwise the children will exist in the system as a zombie and taking up some kernel resources and consume a slot in the kernel process table.

PID 1 in containers

Containers is a concept that isolate processes in different namespaces. Example of such namespaces are PID, users, networking and filesystem. To create a container is quite simple, just create a new process with clone(2) and provide relevant flags to create new namespaces for the process.

The flags related to namespaces are listed in include/uapi/linux/sched.h:

#define CLONE_NEWPID                0x20000000      // New pid namespace
#define CLONE_NEWCGROUP             0x02000000      // New cgroup namespace
#define CLONE_NEWUTS                0x04000000      // New utsname namespace
#define CLONE_NEWIPC                0x08000000      // New ipc namespace
#define CLONE_NEWUSER               0x10000000      // New user namespace
#define CLONE_NEWPID                0x20000000      // New pid namespace
#define CLONE_NEWNET                0x40000000      // New network namespace

All processes is running in a "container-context" because the processes allways executes in a namespace. On a system "without containers", all processes still have one common namespace that all processes is using.

When using CLONE_NEWPID, the kernel will create a new PID namespace and let the newly created process has the PID 1. As we already know, the PID 1 process has a very special task, namely to kill all orphaned children. This PID 1 process could be any application (make, bash, nginx, ftp-server or whatever) that is missing this essential adopt-and-slay-mechanism. If the reaping is not handled, it will result in zombie-processes. This was a real problem not long time ago for Docker containers (google Docker and zombies to see what I mean). Nowadays we have the --init flag on docker run to tell the container to use tini (https://github.com/krallin/tini), a zombie-reaping init process to run with PID 1.

When PID 1 dies

This is the reason to why I'm writing this post. I was wondering who is killing PID 1 in a container since we learned that a PID 1 may not die under any circumstances. PID 1 in cointainers is obviosly an exception from this golden rule, but how does the kernel differentiate between init processes in different PID namespaces?

Lets follow a process to its very last breath.

The call chain we will look at is the following: do_exit()->exit_notify()->forget_original_parent()->find_child_reaper().

......

<<<<< Collect exit code >>>>>
tsk->exit_code = code;
taskstats_exit(tsk, group_dead);


exit_mm(tsk);

if (group_dead)
    acct_process();
trace_sched_process_exit(tsk);

<<<<< Free up resources >>>>>
exit_sem(tsk);
exit_shm(tsk);
exit_files(tsk);
exit_fs(tsk);
if (group_dead)
    disassociate_ctty(1);
exit_task_namespaces(tsk);
exit_task_work(tsk);
exit_thread(tsk);

perf_event_exit_task(tsk);

sched_autogroup_exit_task(tsk);
cgroup_exit(tsk);

<<<<< Notify tasks in the same group >>>>>
exit_notify(tsk, group_dead);

.........

\*\*
\* A maximum of 4 million PIDs should be enough for a while.
\* [NOTE: PID/TIDs are limited to 2^29 ~= 500+ million, see futex.h.]
\*/
#define PID_MAX_LIMIT (CONFIG_BASE_SMALL ? PAGE_SIZE * 8 : \
(sizeof(long) > 4 ? 4 * 1024 * 1024 : PID_MAX_DEFAULT))

static void exit_notify(struct task_struct *tsk, int group_dead)
{
    bool autoreap;
    struct task_struct *p, *n;
    LIST_HEAD(dead);

    write_lock_irq(&tasklist_lock);
  >>>>>  forget_original_parent(tsk, &dead);

>>>>> reaper = find_child_reaper(father);
if (list_empty(&father->children))
    return;

if (unlikely(pid_ns == &init_pid_ns)) {
    panic("Attempted to kill init! exitcode=0x%08x\n",
        father->signal->group_exit_code ?: father->exit_code);
}
zap_pid_ns_processes(pid_ns);

nr = next_pidmap(pid_ns, 1);
while (nr > 0) {

    rcu_read_lock();

    task = pid_task(find_vpid(nr), PIDTYPE_PID);
    if (task && !__fatal_signal_pending(task))
    >>>>> send_sig_info(SIGKILL, SEND_SIG_FORCED, task);

    rcu_read_unlock();

    nr = next_pidmap(pid_ns, nr);

}

Conclusion

The PID 1 in containers is handled in a seperate way than the real init process. This is obvious, but now we know where the codeflow differ for PID 1 in different namespaces.

We also see that if the PID1 in a PID namespace dies, all the subprocesses will be terminated with SIGKILL. This behavior reflects the fact that the init process is essential for the correct operation of any PID namespace.

Boot up the BeagleBone

I haven't booted up my bone for a while and support for the board seems to have reached the mainline in v3.8 (currently at v3.15), so the first step is just to get it boot with a custom kernel.

Clone the vanilla kernel from kernel.org:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Use the omap2plus_defconfig as base:

make ARCH=arm omap2plus_defconfig

I will still use my old U-boot version, which does not have support for devicetrees, so I have to make sure that

CONFIG_ARM_APPENDED_DTB=y

This simply tells the boot code to look for a device tree binary (DTB) appended to the zImage. Without this option, the kernel expects the address of a dtb in the r2 register (on ARM architectures), but that does not work on my ancient bootloader.

Next step is to compile the kernel. We are using U-Boot as bootloader, but we do not create an uImage since we have to append the dtb to the zImage before that.:

make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi-

Next, create the device tree blob. We are using the arch/arm/dts/am335x-bone.dts as source.:

make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- am33x-bone.dtb

Now we are only two steps behind a booting kernel! First we need to append the dtb to the zImage, and then we need to create an U-boot-friendly kernel image with mkimage.:

cat arch/arm/boot/zImage arch/arm/boot/dts/am335x-bone.dtb > ./zImage_dtb
mkimage -A arm -O linux -T kernel -C none -a 0x80008000 -e 0x80008000 -n 'BeagleBone image' -d ./zImage_dtb uImage

Put the uImage on the uSD-card and boot it up. ..

BeagleBone login:

Victory!

Enable SPI

First of all, we need to setup the pinmux for the spi-bus. This is done with the pinctrl subsystem in the devicetree interface file (arch/arm/boot/dts/am335x-bone-common.dtsi).

Create the pins. For more detailed explaination of the values, see the BeagleBone System Reference Manual.

spi1_pins: spi1_pins_s0 {
   pinctrl-single,pins = <
     0x190 0x33      /* mcasp0_aclkx.spi1_sclk, INPUT_PULLUP | MODE3 */
     0x194 0x33      /* mcasp0_fsx.spi1_d0, INPUT_PULLUP | MODE3 */
     0x198 0x13      /* mcasp0_axr0.spi1_d1, OUTPUT_PULLUP | MODE3 */
     0x19c 0x13      /* mcasp0_ahclkr.spi1_cs0, OUTPUT_PULLUP | MODE3 */
 >;
};

Then override the spi1 entry and create an instance of our device driver. The driver will have the name "ili9341-fb".

&spi1{
 status = "okay";
 pinctrl-names = "default";
 pinctrl-0 = <&spi1_pins>;
 ili9341: ili9341@0 {
  compatible = "ili9341-fb";
  reg = <0>;
  spi-max-frequency = <16000000>;
  dc-gpio = <&gpio3 19 GPIO_ACTIVE_HIGH>;
 };
};

Create an entry in the Kbuild system

I always integrate the modules into the kbuild system as the first step. This for several reasons: - I use one kernel for all of my projects, just different branches - It is simple to jump around with cscope/ctags - It gives you control when the kernel version and your driver follow eachother - Out-of-tree modules is evil (gives you a tainted kernel and everyone will spit on you)

Those who don't know how to put a module into the kbuild system - get ready to be surprised how simple it is!

Every directory in the kernel structure contains at least two files, a Makefile and a Kconfig. The Makefile tells the make buildsystem which files to compile and the Kconfig file is interpreted by (menu|k|x|old|....)config.

Here is what's needed:

diff --git a/drivers/video/fbdev/Kconfig b/drivers/video/fbdev/Kconfig
index e1f4727..be4ec8f 100644
--- a/drivers/video/fbdev/Kconfig
+++ b/drivers/video/fbdev/Kconfig
@@ -163,6 +163,18 @@ config FB_DEFERRED_IO
        bool
        depends on FB
+config FB_ILI9341
+       tristate "ILI9341 TFT driver"
+       depends on FB
+       select FB_SYS_FILLRECT
+       select FB_SYS_COPYAREA
+       select FB_SYS_IMAGEBLIT
+       select FB_SYS_READ
+       select FB_DEFERRED_IO
+       ---help---
+       This enables functions for handling video modes using the ili9341 controller
+
 config FB_HECUBA
        tristate
        depends on FB
diff --git a/drivers/video/fbdev/Makefile b/drivers/video/fbdev/Makefile
index 0284f2a..105166a 100644
--- a/drivers/video/fbdev/Makefile
+++ b/drivers/video/fbdev/Makefile
@@ -60,6 +60,7 @@ obj-$(CONFIG_FB_ATARI)            += atafb.o c2p_iplan2.o atafb_mfb.o \
                                      atafb_iplan2p2.o atafb_iplan2p4.o atafb_iplan2p8.o
 obj-$(CONFIG_FB_MAC)              += macfb.o
 obj-$(CONFIG_FB_HECUBA)           += hecubafb.o
+obj-$(CONFIG_FB_ILI9341)          += ili9341.o
 obj-$(CONFIG_FB_N411)             += n411.o
 obj-$(CONFIG_FB_HGA)              += hgafb.o
 obj-$(CONFIG_FB_XVR500)           += sunxvr500.o
diff --git a/drivers/video/fbdev/ili9341.c b/drivers/video/fbdev/ili9341.c

Deferred IO

Deferred IO is a way to delay and repurpose IO. It uses host memory as a buffer and the MMU pagefault as a pretrigger for when to perform the device IO. You simple tell the kernel the minimum delay between the triggers should occours, this allows you to do burst transfers to the device at a given framerate. This has the big benefit that if the userspace updates the framebuffer several times in this period, we will only write it once.

The interface is _really_ simple. All you need to follow is these four steps (see Documentation/fb/deferred_io.txt):

1. Setup your structure.

   static struct fb_deferred_io hecubafb_defio = {
    .delay  = HZ,
    .deferred_io = hecubafb_dpy_deferred_io,
   };
   

The delay is the minimum delay between when the page_mkwrite trigger occurs and when the deferred_io callback is called. The deferred_io callback is explained below.

2. Setup your deferred IO callback.

   static void hecubafb_dpy_deferred_io(struct fb_info *info,
       struct list_head *pagelist)
   

The deferred_io callback is where you would perform all your IO to the display device. You receive the pagelist which is the list of pages that were written to during the delay. You must not modify this list. This callback is called from a workqueue.

3. Call init:

   info->fbdefio = &hecubafb_defio;
   fb_deferred_io_init(info);
   

4. Call cleanup:

   fb_deferred_io_cleanup(info);
   

Problems

The driver is quite straight forward and there was no really hard problem with the driver itself. However, I had problem to get a high framerate because the SPI communication took time. All SPI communication is asynchronious and all jobs is stacked on a queue before it gets scheduled. This takes time. One obvious solution is to write bigger chunks with each transfer, and that is what I did.

But the problem was that when I increased the chunk size, the kernel got panic with the DMA transfers. After an half a hour of code-digging, the problem is derived to the spi-controller for the omap2 (drivers/spi/spi-omap2-mcspi.c). It defines the DMA_MIN_BYTES which is arbitrarily set to 160. The code then compare the data length to this constant and determine if it should use DMA or not. It shows up that the DMA-transfer-code itself is broken.

A temporary solution is to increase the DMA_MIN_BYTES to at least a full frame (240x320x2) bytes until I have looked at the DMA code and submitted a fix :-)

Result

Here is a shell started from Ubuntu

I have also tested to startup Qt and directfb applications. It all works like a charm. Conclusion

The Deferred IO interface is really nice for such displays. I'm surprised that there is currently so few drivers using it.

(the not so cleaned up) Code:

/\*
 \* linux/drivers/video/ili9341.c -- FB driver for ili9341 controller
 \*
 \* Copyright (C) 2014, Marcus Folkesson
 \*
 \* This file is subject to the terms and conditions of the GNU General Public
 \* License. See the file COPYING in the main directory of this archive for
 \* more details.
 \*
 \*/
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/errno.h>
#include <linux/string.h>
#include <linux/mm.h>
#include <linux/vmalloc.h>
#include <linux/delay.h>
#include <linux/interrupt.h>
#include <linux/fb.h>
#include <linux/init.h>
#include <linux/list.h>
#include <linux/uaccess.h>
#include <linux/spi/spi.h>
#include <video/ili9341.h>
#include <linux/regmap.h>
#include <linux/gpio.h>
#include <linux/of.h>
#include <linux/gpio.h>
#include <linux/of_gpio.h>
#include <linux/debugfs.h>

/\* Display specific information \*/
#define SCREEN_WIDTH (240)
#define SCREEN_HIGHT (320)
#define SCREEN_BPP  (16)
#define ID "cbdff49d683b"
#define ID_SZ 12

static unsigned int chunk_size;

struct ili9341_priv {
 struct spi_device *spi;
 struct regmap *regmap;
 struct fb_info *info;
 u32 vsize;
 int dc;
 char *fbmem;
 struct dentry *dir;
};
static struct fb_fix_screeninfo ili9341_fix = {
 .id =  "ili9341",
 .type =  FB_TYPE_PACKED_PIXELS,
 /*.visual = FB_VISUAL_MONO01,*/
 .visual = FB_VISUAL_PSEUDOCOLOR,
 .xpanstep = 0,
 .ypanstep = 0,
 .ywrapstep = 0,
 .line_length = SCREEN_WIDTH*2,
 .accel = FB_ACCEL_NONE,
};
static struct fb_var_screeninfo ili9341_var = {
 .xres  = SCREEN_WIDTH,
 .yres  = SCREEN_HIGHT,
 .xres_virtual = SCREEN_WIDTH,
 .yres_virtual = SCREEN_HIGHT,
 .bits_per_pixel = SCREEN_BPP,
 .nonstd  = 1,
 .red = {
  .offset = 11,
  .length = 5,
 },
 .green = {
  .offset = 5,
  .length = 6,
 },
 .blue = {
  .offset = 0,
  .length = 5,
 },
 .transp = {
  .offset = 0,
  .length = 0,
 },
};


static const struct regmap_config ili9341_regmap_config = {
 .reg_bits = 8,
 .val_bits = 8,
 .can_multi_write = 1,
};
static void fill(struct ili9341_priv *priv);
static void fill_area(struct ili9341_priv *priv, int y1, int y2);
/* main ili9341 functions */
static void apollo_send_data(struct ili9341_priv *par, unsigned char data)
{
 return;
 /* set data */
}
static void apollo_send_command(struct ili9341_priv *par, unsigned char data)
{
 return;
}
static void ili9341_dpy_update(struct ili9341_priv *par)
{
 /*return;*/
 fill(par);
}
static void ili9341_dpy_update_area(struct ili9341_priv *par, int y1, int y2 )
{
 /*return;*/
 fill_area(par, y1, y2);
}
/* this is called back from the deferred io workqueue */
static void ili9341_dpy_deferred_io(struct fb_info *info,
    struct list_head *pagelist)
{
 struct page *cur;
 struct fb_deferred_io *fbdefio = info->fbdefio;
 struct ili9341_priv *par = info->par;

 struct page *page;
 unsigned long beg, end;
 int y1, y2, miny, maxy;
 miny = INT_MAX;
 maxy = 0;
 /* stop here if list is empty */
 if (list_empty(pagelist)){
  dev_err(&par->spi->dev, "pagelist is empty");
  return;
 }
 list_for_each_entry(page, pagelist, lru) {
  beg = page->index << PAGE_SHIFT;
  end = beg + PAGE_SIZE - 1;
  y1 = beg / (info->fix.line_length);
  y2 = end / (info->fix.line_length);
  if (y2 >= info->var.yres)
   y2 = info->var.yres - 1;
  if (miny > y1)
   miny = y1;
  if (maxy < y2)
   maxy = y2;
   }
 ili9341_dpy_update_area(info->par, miny, maxy);
 // dev_err(&par->spi->dev, ".");
}
static void ili9341_fillrect(struct fb_info *info,
       const struct fb_fillrect *rect)
{
 struct ili9341_priv *par = info->par;
 sys_fillrect(info, rect);
 /*ili9341_dpy_update(par);*/
}
static void ili9341_copyarea(struct fb_info *info,
       const struct fb_copyarea *area)
{
 struct ili9341_priv *par = info->par;
 sys_copyarea(info, area);
 /*ili9341_dpy_update(par);*/
}
static void ili9341_imageblit(struct fb_info *info,
    const struct fb_image *image)
{
 struct ili9341_priv *par = info->par;
 sys_imageblit(info, image);
 /*ili9341_dpy_update(par);*/
}
/*
 * this is the slow path from userspace. they can seek and write to
 * the fb. it's inefficient to do anything less than a full screen draw
 */
static ssize_t ili9341_write(struct fb_info *info, const char __user *buf,
    size_t count, loff_t *ppos)
{
 struct ili9341_priv *par = info->par;
 unsigned long p = *ppos;
 void *dst;
 int err = 0;
 unsigned long total_size;
 if (info->state != FBINFO_STATE_RUNNING)
  return -EPERM;
 total_size = info->fix.smem_len;
 if (p > total_size)
  return -EFBIG;
 if (count > total_size) {
  err = -EFBIG;
  count = total_size;
 }
 if (count + p > total_size) {
  if (!err)
   err = -ENOSPC;
  count = total_size - p;
 }
 dst = (void __force *) (info->screen_base + p);
 if (copy_from_user(dst, buf, count))
  err = -EFAULT;
 if  (!err)
  *ppos += count;
 ili9341_dpy_update(par);
 return (err) ? err : count;
}
static struct fb_ops ili9341_ops = {
 .owner   = THIS_MODULE,
 .fb_write  = ili9341_write,
 .fb_fillrect = ili9341_fillrect,
 .fb_copyarea = ili9341_copyarea,
 .fb_imageblit = ili9341_imageblit,
};
static struct fb_deferred_io ili9341_defio = {
 .delay  = HZ/60,
 .deferred_io = ili9341_dpy_deferred_io,
};

static void write_command(struct ili9341_priv *priv, u8 data)
{
 gpio_set_value(priv->dc, 0);
 spi_write(priv->spi, &data, 1);
 gpio_set_value(priv->dc, 1);
}
static void write_data(struct ili9341_priv *priv, u8 data)
{
 gpio_set_value(priv->dc, 1);
 spi_write(priv->spi, &data, 1);
}
static void write_data16(struct ili9341_priv *priv, u8 data)
{
 gpio_set_value(priv->dc, 1);
 spi_write(priv->spi, &data, 1);
}
static void init(struct ili9341_priv *priv)
{
 write_command(priv, 0xCB);
 write_data(priv, 0x39);
 write_data(priv, 0x2C);
 write_data(priv, 0x00);
 write_data(priv, 0x34);
 write_data(priv, 0x02);
 write_command(priv, 0xCF);
 write_data(priv, 0x00);
 write_data(priv, 0XC1);
 write_data(priv, 0X30);
 write_command(priv, 0xE8);
 write_data(priv, 0x85);
 write_data(priv, 0x00);
 write_data(priv, 0x78);
 write_command(priv, 0xEA);
 write_data(priv, 0x00);
 write_data(priv, 0x00);
 write_command(priv, 0xED);
 write_data(priv, 0x64);
 write_data(priv, 0x03);
 write_data(priv, 0X12);
 write_data(priv, 0X81);
 write_command(priv, 0xF7);
 write_data(priv, 0x20);
 write_command(priv, 0xC0);     //Power control
 write_data(priv, 0x23);    //VRH[5:0]
 write_command(priv, 0xC1);     //Power control
 write_data(priv, 0x10);    //SAP[2:0];BT[3:0]
 write_command(priv, 0xC5);     //VCM control
 write_data(priv, 0x3e);    //Contrast
 write_data(priv, 0x28);
 write_command(priv, 0xC7);     //VCM control2
 write_data(priv, 0x86);    //--
/* XXX: Hue?! */
 write_command(priv, 0x36);     // Memory Access Control
 write_data(priv, 0x48);   //C8    //48 68绔栧睆//28 E8 妯睆
 write_command(priv, 0x3A);
 write_data(priv, 0x55);
 write_command(priv, 0xB1);
 write_data(priv, 0x00);
 write_data(priv, 0x18);
 write_command(priv, 0xB6);     // Display Function Control
 write_data(priv, 0x08);
 write_data(priv, 0x82);
 write_data(priv, 0x27);

 write_command(priv, 0xF2);     // 3Gamma Function Disable
 write_data(priv, 0x00);
 write_command(priv, 0x26);     //Gamma curve selected
 write_data(priv, 0x01);
 write_command(priv, 0xE0);     //Set Gamma
 write_data(priv, 0x0F);
 write_data(priv, 0x31);
 write_data(priv, 0x2B);
 write_data(priv, 0x0C);
 write_data(priv, 0x0E);
 write_data(priv, 0x08);
 write_data(priv, 0x4E);
 write_data(priv, 0xF1);
 write_data(priv, 0x37);
 write_data(priv, 0x07);
 write_data(priv, 0x10);
 write_data(priv, 0x03);
 write_data(priv, 0x0E);
 write_data(priv, 0x09);
 write_data(priv, 0x00);
 write_command(priv, 0XE1);     //Set Gamma
 write_data(priv, 0x00);
 write_data(priv, 0x0E);
 write_data(priv, 0x14);
 write_data(priv, 0x03);
 write_data(priv, 0x11);
 write_data(priv, 0x07);
 write_data(priv, 0x31);
 write_data(priv, 0xC1);
 write_data(priv, 0x48);
 write_data(priv, 0x08);
 write_data(priv, 0x0F);
 write_data(priv, 0x0C);
 write_data(priv, 0x31);
 write_data(priv, 0x36);
 write_data(priv, 0x0F);
 write_command(priv, 0x11);     //Exit Sleep
 mdelay(100);
 write_command(priv, 0x29);    //Display on
 write_command(priv, 0x2c);
}
static void setCol(struct ili9341_priv *priv, u16 start, u16 end)
{
 u8 tmp;
 write_command(priv, 0x2a);
 tmp = (start & 0xff00) >> 8;
 write_data(priv, tmp);
 tmp = (start & 0x00ff) >> 0;
 write_data(priv, tmp);

 tmp = (end & 0xff00) >> 8;
 write_data(priv, tmp);
 tmp = (end & 0x00ff) >> 0;
 write_data(priv, tmp);
}
static void setPage(struct ili9341_priv *priv, u16 start, u16 end)
{
 u8 tmp;
 write_command(priv, 0x2b);
 tmp = (start & 0xff00) >> 8;
 write_data(priv, tmp);
 tmp = (start & 0x00ff) >> 0;
 write_data(priv, tmp);

 tmp = (end & 0xff00) >> 8;
 write_data(priv, tmp);
 tmp = (end & 0x00ff) >> 0;
 write_data(priv, tmp);
}
static void setPos(struct ili9341_priv *priv, u16 x1, u16 x2, u16 y1, u16 y2)
{
 setPage(priv, y1, y2);
 setCol(priv, x1, x2);
}

static void fill_area(struct ili9341_priv *priv, int y1, int y2)
{
 int i = 0;
 char val = 0xaa;
 char *p = priv->fbmem;
 int ret;
 int start =y1*SCREEN_WIDTH*2 + 1;
 int stop = y2*SCREEN_WIDTH*2+1;
 int range = stop - start;

 if (!chunk_size)
  chunk_size = 10;
 if (start + range > priv->vsize)
  range = priv->vsize - start;
 setCol(priv, 0, 239);
 setPage(priv, y1, y2);
 write_command(priv, 0x2c);


 for(i = start; i < stop; i += chunk_size)
 {
  if ( i + chunk_size > stop )
   chunk_size = stop - i;
  ret = spi_write(priv->spi, &priv->fbmem[i], chunk_size);
  if (ret != 0)
   dev_err(&priv->spi->dev, "Error code: %i\n", ret);
   }
}
static void fill(struct ili9341_priv *priv)
{
 int i = 0;
 char val = 0xaa;
 char *p = priv->fbmem;
 setCol(priv, 0, 239);
 setPage(priv, 0, 319);
 write_command(priv, 0x2c);

 fill_area(priv, 0, 319);
}
static ssize_t id_show(struct device *dev, struct device_attribute *attr,
   char *buf)
{
 sprintf(buf, "%s", ID);
 return ID_SZ;
}
static ssize_t id_store(struct device *dev, struct device_attribute *attr,
    const char *buf, size_t count)
{
 char kbuf[ID_SZ];
 if (count != ID_SZ)
  return -EINVAL;
 memcpy(kbuf, buf, ID_SZ);
 if (memcmp(kbuf, ID, ID_SZ) != 0)
  return -EINVAL;
 return count;
}
DEVICE_ATTR(id, 0666, id_show, id_store);
static struct attribute *ili9341_attrs[] = {
 &dev_attr_id.attr,
 NULL,
};
ATTRIBUTE_GROUPS(ili9341);


static int ili9341_probe(struct spi_device *spi)
{
 struct fb_info *info;
 int retval = -ENOMEM;
 struct ili9341_priv *priv;
 struct device_node *np = spi->dev.of_node;
 int ret;
 dev_err(&spi->dev, "Hello from I!\n");
 priv = kzalloc(sizeof(struct ili9341_priv), GFP_KERNEL);
 if(!priv)
  return -ENOMEM;
 priv->spi = spi;

/* TODO: better fail handling... */
 priv->dc = of_get_named_gpio(np, "dc-gpio", 0);
 if (priv->dc  == -EPROBE_DEFER)
  return -EPROBE_DEFER;
 if (gpio_is_valid(priv->dc)) {
  ret = devm_gpio_request(&spi->dev, priv->dc, "tft dc");
  if (ret)
   dev_err(&spi->dev, "could not request dc\n");
  ret = gpio_direction_output(priv->dc, 1);
  if (ret)
   dev_err(&spi->dev, "could not set DC to output");
 }else
  dev_err(&spi->dev, "DC gpio is not valid");


 dev_err(&spi->dev, "Initialize regmap");
 priv->regmap = devm_regmap_init_spi(spi, &ili9341_regmap_config);
 if (IS_ERR(priv->regmap))
  goto err_regmap;
 dev_err(&spi->dev, "regmap OK");
 priv->vsize = (SCREEN_WIDTH*SCREEN_HIGHT)*(SCREEN_BPP/8);
 priv->fbmem = vzalloc(priv->vsize);
 if (!priv->fbmem)
  goto err_videomem_alloc;

 init(priv);
 fill(priv);
 retval = sysfs_create_group(&spi->dev.kobj, *ili9341_groups);
 if (retval)
  kobject_put(&spi->dev.kobj);


 dev_err(&spi->dev, "Allocate framebuffer");
 info = framebuffer_alloc(sizeof(struct fb_info), &spi->dev);
 if (!info)
  goto err_fballoc;

 info->par = priv;
 priv->info = info;
 info->screen_base = priv->fbmem;
 info->fbops = &ili9341_ops;
 info->var = ili9341_var;
 info->fix = ili9341_fix;
 info->fix.smem_len = priv->vsize;
 /* We are virtual as we only exists in memory */
 info->flags = FBINFO_FLAG_DEFAULT | FBINFO_VIRTFB;
 info->fbdefio = &ili9341_defio;
 fb_deferred_io_init(info);
 retval = register_framebuffer(info);
 if (retval < 0)
  goto err_fbreg;
 spi_set_drvdata(spi, info);
 fb_info(info, "Hecuba frame buffer device, using %dK of video memory\n",
  priv->vsize >> 10);



 priv->dir = debugfs_create_dir("ili9341-fb", NULL);
 debugfs_create_u32("chunk_size", 0666, priv->dir, &chunk_size);

 return 0;
err_fbreg:
 framebuffer_release(info);
err_fballoc:
 vfree(priv->fbmem);
err_videomem_alloc:
err_regmap:
 kfree(priv);
 return retval;
}
static int ili9341_remove(struct spi_device *spi)
{
 struct fb_info *info = spi_get_drvdata(spi);
 if (info) {
  struct ili9341_priv *priv = info->par;
  fb_deferred_io_cleanup(info);
  unregister_framebuffer(info);
  vfree(info->screen_base);
  framebuffer_release(info);
  kfree(priv);
 }
 return 0;
}

static const struct spi_device_id ili9341_ids[] = {
 {"ili9341-fb", 0},
 {}
};
MODULE_DEVICE_TABLE(spi, ili9341_ids);
static struct spi_driver  ili9341_driver = {
 .probe = ili9341_probe,
 .remove = ili9341_remove,
 .id_table = ili9341_ids,
 .driver = {
  .owner = THIS_MODULE,
  .name = "ili9341-fb",
 },
};
module_spi_driver(ili9341_driver);
MODULE_DESCRIPTION("fbdev driver for ili9341 controller");
MODULE_AUTHOR("Marcus Folkesson <marcus.folkesson@gmail.com>");
MODULE_LICENSE("GPL");

Setting up a timer

The usage of the hrtimers is really simple.

1. Initialize a struct hrtimer with

   hrtimer_init(struct hrtimer *timer, clockid_t which_clock, enum hrtimer_mode mode);
   

timer is a pointer to the instance of the struct hrtimer. clock is the clock to bind the timer to, often CLOCK_MONOTONIC or CLOCK_REALTIME. mode specifies if the timer is working with absolute or relative time values. Two constants are available: HRTIMER_MODE_ABS and HRTIMER_MODE_REL.

Set a callback function with:

mytimer.function = my_callback;

Where my_callback is declared as:

enum hrtimer_restart my_callback(struct hrtimer *timer)

Start the timer with hrtimer_start:

struct ktime_t delay = ktime_set(5,0);
hrtimer_start(&mytimer, delay, CLOCK_MONOTONIC);

ktime_set initialize delay with 5 seconds and 0 nanoseconds.

Wait. The callback function will be called after 5s!

A full example

struct hrtimer mytimer;
ktime_t delay = ktime_set(5, 0);

enum hrtimer_restart my_callback(struct hrtimer *timer)
{
    printk("Hello from timer!\n");
    return HRTIMER_NORESTART;
}

void ....()
{
    hrtimer_init(&mytimer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
    mytimer.function = my_callback;

    hrtimer_start(&mytimer, delay, CLOCK_MONOTONIC);
}

Further reading

There is more functions related to the hrtimers. See incude/linux/hrtimer.h for a full list. Other useful functions are:

int hrtimer_cancel(struct hrtimer *timer)
int hrtimer_try_to_cancel(struct hrtimer *timer)
int hrtimer_restart(struct hrtimer *timer)