Interrupts, and how to divide them between cores

Symetric MultiProcessing (SMP) are becoming increasingly common in embedded systems. Many chip manufacturers such as Texas Instruments and Freescale offers ARM multicores, even FPGA manufacturers like Xilinx and Altera has System-on-Chip with multiple ARM cores. One benefit with SoC is that it is even possible to add soft cores in the programmable logic if it's necessary.

The trend is clear, multiple cores is here and it is not likely to be fewer of them.

But how do we share resources between the cores? One way is to use cgroups (Control Groups) that was merged into the kernel version 2.6.24. Cgroups let you divide resources like cores and memory to specific task groups. More about that in another post.

It is also possible to tell which processor core that is allowed to handle a specific interrupt. This is done in procfs. There are a few entries in procfs related to interrupt-handling. The first interesting entry is /proc/interrupts that is used to record the number of interrupts per CPU. Beside the the counts of hard IRQ numbers, it also includes interrupts internal to the system that is not associated with a device. Examples on these are NMI (nonmaskable interrupts) and LOC (Local timer interrupt). It also gives us the name of the registred IRQ handler.

Example output:

SMP affinity is controlled by manipulating files in the /proc/irq directory. In /proc/irq is the file default_smp_affinity that specifies the default affinity mask that applies to all non-active IRQs. Once the IRQ is allocated/activeded its affinity bitmask will be set to the default mask. The default mask is set so all available cpu cores is allowed to handle the interrupt.

In /proc/irq is also directories that correspond to the IRQ present in the system. These directories has a few entries where two of them are of interest.

Here is the entries for IRQ 150, which correspond to the ethernet controller (look at the output from /proc/interrupts): # ls /proc/irq/150 affinity_hint smp_affinity spurious node smp_affinity_list smp_affinity_list contains the available processor cores that are able to handle the interrupt, and the smp_affinity is the bitmask of processor cores that is allowed to handle it.

The current values are:

This tells us that processor core nr 0 and 1 is able to handle the interrupt and processor 0 and 1 (bit 0 and 1 gives us 3 dec) is allowed to handle it. If we look at the output from /proc/interrupts, we see that all interrupts (19293) has been handled by the CPU0. If we just want CPU1 to handle these interrupts, set smp_affinty to 2:

# echo 2 > /proc/irq/150/smp_affinity

Then look at /proc/interrupts again:

# cat /proc/interrupts
           CPU0       CPU1
150:      20967         12       GIC  enet

The CPU1 interrupt counter is ticking and the interrupt is now only served by CPU1.