Changing the root of your Linux filesystem

After my previous post [1], I've got a few questions from people about the difference between chroot, pivot_root and switch_root, they all seems to do the same thing, right? Almost.

Lets shed some light on this topic.

Rootfs

First of all, we need to specify what we mean when we say rootfs.

rootfs is a special instance of a ram filesystem (usually tpmfs) that is created in an early boot stage. You cannot unmount rootfs for pretty much the same reason you can't kill the init process - you need it to have a functional system.

The kernel build process always creates a gzipped gpio (newc format) archive and ink it into the resulting kernel binary. This is our rootfs! The empty archive consuming ~130 bytes, and that is something we have to live with.

The rootfs is empty - unless you populate it. You may set the CONFIG_INITRAMFS_SOURCE config option to include files and directories to your rootfs.

When the system boots, it searches for an executable /init in the rootfs, and executes if it exists. If it does not exist the system will continue and try to mount whatever is specified on the root= kernel command line.

One important thing to remember is that the system mounts the root= partition on / (which is basically rootfs).

initramfs vs inird

So, now we when we know what rootfs is and that rootfs is basically an initramfs (empty or not).

Now we need to distinguish between initramfs and initrd. At a first glance, it may seems like they're the same, but they are not.

An initrd is basically a complete image of a filesystem that is mounted on a ramdev, a RAM-based block-device. As it's a filesystem, you will need to have support for that filesystem static compiled (=y) into your kernel.

An initramfs, on the other hand, is not a filesystem. It's a cpio archive which is unpacked into a tmpfs.

As we already has spoken about, this archive is embedded into the kernel itself and as a side-effect, the initramfs could be loaded earlier in the kernel boot process and keep a smaller footprint as the kernel can adapt the size of the tmpfs to what is actually loaded.

The both are also fundamental different when it comes how they're treated as a root filesystem. As an initrd is an actual filesystem unpacked into RAM, the root filesystem will be that ramdisk. When changing root filesystem form an initrd to a "real" root device, you have to switch the root device.

The initramfs on the other hand, is some kind of interim filesystem with no underlaying device, so the kernel doesn't need to switch any devices - just "jump" to one.

Jump into a new root filesystem

So, when it's time to "jump" to a new root filesystem, that could be done in several ways.

chroot

chroot runs a command with a specified root directory. Nothing more, nothing less.

The basic syntax of the chroot command is: 

chroot option newroot [command [args]…]

pivot_root

pivot_root moves the root file system of the current process to the directory put_old and makes new_root the new root file system. It's often used in combination with chroot.

From the manpage [2]:

The typical use of pivot_root() is during system startup,
when the system mounts a temporary root file
system (e.g., an initrd), then mounts the real root file system,
and eventually turns the latter into the current root
of all relevant processes or threads.

The basic syntax of the pivot_root command is: 

pivot_root new_root put_old

Example on how to use pivot_root:

mount /dev/hda1 /new-root
cd /new-root
pivot_root . old-root
exec chroot . sh <dev/console >dev/console 2>&1
umount /old-root

pivot_root should be unused for initrd only, it does not work for initramfs.

switch_root

switch_root [3] switches to another filesystem as the root of the mount tree. Pretty much the same thing as pivot_root do, but with a few differences.

switch_root moves already mounted /proc, /dev, /sys and /run to newroot and makes newroot the new root filesystem and starts init process.

Another important difference is that switch_root removes recursively all files and directory in the current root filesystem!

The basic syntax of the switch_root command is: 

switch_root newroot init [arg...]

Also, you are not allowed to umount the old mount (as you do with pivot_root) as this is your rootfs (remember?).

Summary

To summarize;

If you use initrd - use pivot_root combined with chroot.

if you use initramfs - use switch_root.

It's technically possible to only use chroot, but it will not free resources and you could end up with some restrictions( see unshare(7) for example).

Also, keep in mind that you must invoke either command via exec, otherwise the new process will not inherit PID 1.