What to `expect`

When working in Linux, you’ve probably come across commands that require interactive input — things like typing a password or confirming the same prompt over and over again. This can be frustrating when you want to automate tasks or just need to e.g. type the same range of commands every time your embedded Linux system boots up. This is what expect [1] is used for.

So, what is expect?

expect is a tool that automates interactive applications. It works by “expecting” specific outputs and then sending the responses you specify. In other words, it allows you to script interactions with programs that normally require a human to type input.

It doesn't sounds very exciting, does it? It is.

What makes expect so powerful in my daily work is that:

It works independently of the context on the target system. In other words, I can use it to automate interaction with U-Boot, Linux or whatever text stream I'm working with.
It works with serial consoles (!) as well as SSH connections.

It should be added that I'm not an advanced user of expect. Features like multiple branches and pattern matching with regular expressions is something that I know exists and pick up when needed, but it's usually not part of my daily use. My expect scripts are usually quite simple.

Basic usage

Today I needed to flash a board with U-Boot, kernel and rootfs. Usually this is done with JTAG, but I did not have the adapter board with JTAG interface, so the only way to interact with it is via a serial console.

This is something I will do many times a day the coming weeks, so I wrote a simple expect script to automate the process.

 1#!/usr/bin/expect -f
 2
 3# Serial port as first argument
 4set sport [lindex $argv 0]
 5
 6# set the portID and open it for reading and writing
 7set portID [open /dev/$sport r+]
 8set baud 115200
 9
10# Set timeout (default is 10s) to 120s
11set timeout 120
12
13# Configure the port with the baud rate
14# and dont block on read, dont buffer output
15fconfigure $portID -mode "115200,n,8,1"
16fconfigure $portID -blocking 0 -buffering none
17
18spawn -open $portID
19
20expect "Hit any key"
21send "\n"
22expect "u-boot>"
23
24# Set environment variables
25send "setenv serverip 192.168.1.166\n"
26expect "u-boot>"
27send "setenv ipaddr 192.168.1.188\n"
28expect "u-boot>"
29send "setenv bootargs console=ttyS1,115200 ubi.mtd=3,2048 rootfstype=ubifs root=ubi0:rootfs\n"
30expect "u-boot>"
31
32# Flash U-boot
33send "nand erase 0x20000 0x40000 \n"
34expect "u-boot>"
35send "tftp 0xc0000000 u-boot.ais\n"
36expect "u-boot>"
37send "nand write 0xc0000000 0x20000 0x40000\n"
38expect "u-boot>"
39
40# Flash kernel
41send "nand erase 0x100000 0x200000\n"
42expect "u-boot>"
43send "tftp 0xc0000000 uImage\n"
44expect "u-boot>"
45send "nand write 0xc0000000 0x100000 0x200000\n"
46expect "u-boot>"
47
48# Flash rootfs
49send "nand erase 0x300000\n"
50expect "u-boot>"
51send "tftp 0xc2000000 rootfs.ubi\n"
52expect "u-boot>"
53send "nand write 0xc2000000 0x300000 $filesize\n"
54expect "u-boot>"
55
56send "run bootcmd"
57
58# Go into interactive mode and specify exit sequence
59interact {
60    ~.   { send_user "\nQuit signal received\n"; exit }
61}

The script is rather self-explanatory. It opens the serial port, configures it and then spawns an expect session on the port. Then it responds to the expected output by sending new commands. Just a few notes below.

1set sport [lindex $argv 0]

This is how you catch arguments to the script. In this case, the serial port is passed as the first argument. It let you start the scrypt by:

1./my-script ttyUSB0

to make it more flexible.

1set timeout 120

The timeout is default set to 10s. This is not enough as the TFTP transfers can take longer time. So I set it to 120s.

1expect "Hit any key"
2send "\n"
3expect "u-boot>"

This is to break U-boot. The first expect waits for the string "Hit any key" and then sends a newline to break into U-Boot. The next expect waits for the U-Boot prompt.

1interact {
2    ~.   { send_user "\nQuit signal received\n"; exit }
3}

After the flashing is done, I want to interact with the board. The interact command hands over control to the user and let me continue work with the system.

Especially when working with serial consoles, there is no obvious way to exit the interaction mode. So I specify that when I type ~. (tilde followed by a dot), the script should exit. For those who are not familiar with the tilde-dot sequence, it is also used in SSH to break out of the session.

Take it one step further

Okay, I must admit that not all of my expect scripts are that simple. I have a few tips&tricks that I often use.

Subroutines

expect has support for subroutines which is handy when you have a sequence of commands that you want to repeat several times or simply want to run on command.

Consider the following routines:

 1proc flash_kernel {} {
 2    send "tftp 0xc0000000 uImage\n"
 3    expect "u-boot>"
 4    send "nand write 0xc0000000 0x100000 0x200000\n"
 5    expect "u-boot>"
 6} 
 7
 8proc flash_rootfs {} {
 9    send "tftp 0xc2000000 rootfs.ubi\n"
10    expect "u-boot>"
11    send "nand write 0xc2000000 0x300000 0x3200000\n"
12    expect "u-boot>"
13}
14
15proc flash_uboot {} {
16    send "tftp 0xc0000000 u-boot.ais\n"
17    expect "u-boot>"
18    send "nand write 0xc0000000 0x20000 0x40000\n"
19    expect "u-boot>"
20}

Which you may call by their name:

1flash_kernel
2flash_rootfs
3flash_uboot

Combine subroutines with interactive mode

It becomes even more powerful when you combine subroutines with the interactive mode.

Then you can truly automate the parts you want. Consider the following full example:

 1#!/usr/bin/expect -f
 2
 3# Serial port as first argument
 4set sport [lindex $argv 0]
 5
 6proc open_port {} {
 7    global sport
 8
 9    # set the portID and open it for reading and writing
10    set portID [open /dev/$sport r+]
11    set baud 115200
12    set timeout 120
13
14    # Configure the port with the baud rate
15    # and dont block on read, dont buffer output
16    fconfigure $portID -mode "115200,n,8,1"
17    fconfigure $portID -blocking 0 -buffering none
18
19    spawn -open $portID
20
21    # Return the spawn id
22    return $spawn_id
23
24}
25
26proc stop_uboot {} {
27    global spawn_id
28    expect "Hit any key"
29    send "\n"
30    expect "u-boot>"
31}
32
33proc setup_env {} {
34    send "setenv serverip 192.168.1.166\n"
35    expect "u-boot>"
36    send "setenv ipaddr 192.168.1.188\n"
37    expect "u-boot>"
38    send "setenv bootargs console=ttyS1,115200 ubi.mtd=3,2048 rootfstype=ubifs root=ubi0:rootfs\n"
39    expect "u-boot>"
40}
41
42proc flash_kernel {} {
43    send "tftp 0xc0000000 uImage\n"
44    expect "u-boot>"
45    send "nand write 0xc0000000 0x100000 0x200000\n"
46    expect "u-boot>"
47} 
48
49proc flash_rootfs {} {
50    send "tftp 0xc2000000 rootfs.ubi\n"
51    expect "u-boot>"
52    send "nand write 0xc2000000 0x300000 0x3200000\n"
53    expect "u-boot>"
54}
55
56proc flash_uboot {} {
57    send "tftp 0xc0000000 u-boot.ais\n"
58    expect "u-boot>"
59    send "nand write 0xc0000000 0x20000 0x40000\n"
60    expect "u-boot>"
61}
62
63proc reset_board {} {
64    send "reset\n"
65    stop_uboot
66}
67
68proc flash_all {} {
69    setup_env
70    flash_uboot
71    flash_kernel
72    flash_rootfs
73}
74
75# Set $spawn_id to the spawn id of the opened port
76set spawn_id [open_port]
77
78# Stop uboot
79stop_uboot
80
81interact {
82    ~a   { send_user "\nFlash all\n"; flash_all }
83    ~r   { send_user "\nReset board \n"; reset_board }
84    ~.   { send_user "\nQuit signal received\n"; exit }
85}

Here I may just type ~r (tilde followed by r) to reset the board and break into U-Boot, or ~a to flash everything.

Not just for serial consoles

I usually use expect with a serial console, but it works with all applications including SSH, telnet, FTP, etc.

Just replace the spawn command with the application you want to interact with. For example:

1spawn ssh user@example.com

1spawn telnet example.com

Summary

expect is a powerful tool that I really find helpful in my daily work. I pretty much have an expect script for every embedded Linux project I work with.

It helps me remember how to do things on a specific board and I know that my steps are reproducible. The alternative is to have a bunch of scripts with a sequence of commands on target for all the different tasks, but that have several drawbacks:

The scripts will vanish as soon you reflash the system.
The scripts can only be ran in a specific context (e.g. Linux).
The scripts is harder to modify
The scripts may differ on different targets

Of course, expect is not here to replace all your scripts, but it can automate a lot of the repetitive tasks.

Think of, for example, manually setup network interfaces, configure tracefs for a kernel trace session etc.

References

[1]	https://linux.die.net/man/1/expect

What to expect