Interrupts, and how to divide them between cores

Symetric MultiProcessing (SMP) are becoming increasingly common in embedded systems. Many chip manufacturers such as Texas Instruments and Freescale offers ARM multicores, even FPGA manufacturers like Xilinx and Altera has System-on-Chip with multiple ARM cores. One benefit with SoC is that it's even possible to add soft cores in the programmable logic if it's necessary.

The trend is clear, multiple cores is here and it's not likely to be fewer of them.

But how do we share resources between the cores? One way is to use cgroups (Control Groups) that was merged into the kernel version 2.6.24. Cgroups let you divide resources like cores and memory to specific task groups. More about that in another post.

It's also possible to tell which processor core that is allowed to handle a specific interrupt. This is done in procfs. There are a few entries in procfs related to interrupt-handling. The first interesting entry is /proc/interrupts that is used to record the number of interrupts per CPU.

Beside the the counts of hard IRQ numbers, it also includes interrupts internal to the system that is not associated with a device. Examples on these are NMI (nonmaskable interrupts) and LOC (Local timer interrupt). It also gives us the name of the registred IRQ handler.

Example output:

# cat /proc/interrupts
           CPU0       CPU1
 34:          0          0       GIC  sdma
 35:          0          0       GIC  VPU_JPG_IRQ
 37:          0          0       GIC  imx-ipuv3
 38:          0          0       GIC  imx-ipuv3
 39:          0          0       GIC  imx-ipuv3
400:          0          0      GPIO  pfuze
461:          0          0      GPIO  btn volume-up
150:      19293          0       GIC  enet
IPI0:          0       6028  Timer broadcast interrupts
IPI1:       1953      31365  Rescheduling interrupts
IPI2:          0          0  Function call interrupts
IPI3:       1825       1792  Single function call interrupts
IPI4:          0          0  CPU stop interrupts
LOC:      25302        130  Local timer interrupts

SMP affinity is controlled by manipulating files in the /proc/irq directory.

In /proc/irq is the file default_smp_affinity that specifies the default affinity mask that applies to all non-active IRQs. Once the IRQ is allocated/activated its affinity bitmask will be set to the default mask. The default mask is set so all available cpu cores is allowed to handle the interrupt.

In /proc/irq is also directories that correspond to the IRQ present in the system. These directories has a few entries where two of them are of interest; smp_affinity_list and smp_affinity.

Here is the entries for IRQ 150, which correspond to the ethernet controller (look at the output from /proc/interrupts):

# ls -1 /proc/irq/150
affinity_hint
effective_affinity
effective_affinity_list
node
smp_affinity
smp_affinity_list
spurious

smp_affinity_list contains the available processor cores that are able to handle the interrupt, and the smp_affinity is the bitmask of processor cores that is allowed to handle it.

The current values are:

# cat /proc/irq/150/smp_affinity_list
0-1
# cat /proc/irq/150/smp_affinity
3

This tells us that processor core 0 and 1 is able to handle the interrupt and processor 0 and 1 (bit 0 and 1 gives us 3 dec) is allowed to handle it.

If we look at the output from /proc/interrupts, we see that all interrupts (19293) has been handled by the CPU0. If we just want CPU1 to handle these interrupts, set smp_affinty to 2:

# echo 2 > /proc/irq/150/smp_affinity

Then look at /proc/interrupts again:

# cat /proc/interrupts
           CPU0       CPU1
150:      20967         12       GIC  enet

The CPU1 interrupt counter is ticking and the interrupt is now only served by CPU1.