Working with the serial console

From ArchWiki

An Arch Linux machine can be configured for connections via the serial console port, which enables administration of a machine even if it has no keyboard, mouse, monitor, or network attached to it.

Installation of Arch Linux is possible via the serial console as well.

A basic environment for this scenario is two machines connected using a serial cable (9-pin connector cable). The administering machine can be any Unix/Linux or Windows machine with a terminal emulator program (PuTTY or Minicom, for example).

The configuration instructions below will enable boot loader menu selection, boot messages, and terminal forwarding to the serial console.

Configure console access on the target machine

Boot loader

GRUB

When using GRUB with a generated grub.cfg, edit /etc/default/grub and enable serial input and output support: 

/etc/default/grub
...
GRUB_TERMINAL_INPUT="console serial"
...
GRUB_TERMINAL_OUTPUT="gfxterm serial"
...

Next add the GRUB_SERIAL_COMMAND variable and set the options for the serial connection. For COM1 (/dev/ttyS0) with baud rate of 115200 bit/s: 

/etc/default/grub
...
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"

Read GRUB's manual on Using GRUB via a serial line and the serial command for detailed explanation of the available options.

GRUB first stage serial console

When GRUB is installed to an encrypted /boot/ partition - GRUB first stage (core.img) will show Enter passphrase for hdX,gptY: password prompt only on platform console and will not show anything in serial console even if all serial console configuration steps were done properly.

This happens because the grub-install has special behavior for GRUB_ENABLE_CRYPTODISK=y and will create early config placed in /boot/grub/PLATFORM/load.cfg but grub-install ignores serial console configuration from /etc/default/grub for GRUB first stage.

The /boot/grub/PLATFORM/load.cfg file gets overwritten each time grub-install is launched.

To get the GRUB cryptodisk password prompt on the serial console a few extra configuration steps are required:

1. Run the following command to generate /boot/grub/PLATFORM/load.cfg and see the correct grub-mkimage parameters for your system:

EFI platforms: 

# grub-install --verbose 2>&1 | grep grub-mkimage

BIOS platorms: 

# grub-install --verbose /dev/YOUR_BOOT_DISK 2>&1 | grep grub-mkimage

You can reboot the system to check if it boots properly, but you should save the output of the commahd above - it will be hecessary at step 4.

2. Copy /boot/grub/PLATFORM/load.cfg to /boot/grub/PLATFORM/early-grub.cfg

3. Add the following lines at the beginning of /boot/grub/PLATFORM/early-grub.cfg (change those lines according to desired configuration as described above): 

serial --unit=0 --speed=115200
terminal_input serial console
terminal_output serial console

4. Edit grub-mkimage parameters from step 1 (the grub-install output):

Replace --config '/boot/grub/PLATFORM/load.cfg' to --config '/boot/grub/PLATFORM/early-grub.cfg' Remove paramters with empty arguments (like --dtb and --sbat ) and add serial and terminal modules to the end of the grub-mkimage parameters list.

5. Run grub-mkimage with these parameters.

6. On BIOS platforms like i386-pc (for EFI platforms just skip this step) run the following command to install new core.img to your system: 

# grub-bios-setup -d /boot/grub/i386-pc/ /dev/YOUR_BOOT_DISK

In order to revert those changes - just reinstall GRUB using grub-install command.

GRUB Legacy

Edit the GRUB Legacy configuration file /boot/grub/menu.lst and add these lines to the general area of the configuration: 

serial --unit=0 --speed=9600
terminal --timeout=5 serial console
Note: When the terminal --timeout=5 serial console line is added to your menu.lst, your boot sequence will now show a series of Press any key to continue messages. If no key is pressed, the boot menu will appear on whichever (serial or console) appears first in the terminal configuration line.

rEFInd

rEFInd supports serial console only in text mode. Edit refind.conf and uncomment textonly.

Syslinux

To enable serial console in Syslinux, edit syslinux.cfg and add SERIAL as the first directive in the configuration file.

For COM1 (/dev/ttyS0) with baud rate of 115200 bit/s: 

SERIAL 0 115200

The serial parameters are hardcoded to 8 bits, no parity and 1 stop bit.[1]. Read Syslinux Wiki:Config#SERIAL for the directive's options.

Kernel

Kernel's output can be sent to serial console by setting the console= kernel parameter. The last specified console= will be set as /dev/console. 

console=tty0 console=ttyS0,115200

See https://docs.kernel.org/admin-guide/serial-console.html.

getty

At boot, systemd-getty-generator(8) will start a getty instance for each console specified in the kernel command line.

If you have not configured console= in kernel command line start serial-getty@device.service. For /dev/ttyS0 (COM1) that would be serial-getty@ttyS0.service. Enable the service to start it at boot.

Unless specified otherwise in the kernel command line, getty will be expecting 38400 bit/s baud rate, 8 data bits, no parity and one stop bit-times.

Making Connections

Connect using a terminal emulator program

Note: Before making a connection, it is recommended to add yourself to the uucp user group. Otherwise you will need root's permission to make a connection.

Perform these steps on the machine used to connect the remote console.

dterm

dterm-gitAUR is a tiny serial communication program. If you invoke it without parameters, it will connect to /dev/ttyS0 at 9600 baud by default. The following example connect to /dev/ttyS0 at 115200 baud, with 8 data bits, no parity bit and 1 stop bit-times: 

$ dterm 115200 8 n 1

See its README for more examples.

Minicom

minicom can be obtained from the official repositories. Start Minicom in setup mode: 

$ minicom -s

Using the textual navigation menu, change the serial port settings to the following: 

Serial Device: /dev/ttyS0
Bps/Par/Bits: 9600 8N1

Press Enter to exit the menus (pressing Esc will not save changes). Remove the modem Init and Reset strings, as we are not connecting to a modem. To do this, under the Modem and Dialing menu, delete the Init and Reset strings. Optionally save the configuration by choosing save setup as dfl from the main menu. Restart minicom with the serial cable connected to the target machine. The special keys for navigating Minicom can be found in the help menu which is opened by pressing Ctrl+A Z (i.e., exit session is Ctrl+A X).

picocom

picocom is a tiny dumb-terminal emulation program that is very like minicom, but instead of mini, it is pico. The following example connect to ttyS0 at 9600 bps: 

$ picocom -b 9600 /dev/ttyS0
Note: if the backspace key will not work properly try out this option: --omap delbs

See its manual for detailed usage.

Screen

GNU Screen is able to connect to a serial port. It will connect at 9600 baud by default: 

$ screen /dev/ttyS0

A different baud rate (e.g. 115200) may be specified on the command line. 

$ screen /dev/ttyS0 115200

To end the session, press Ctrl+a followed by K. Alternatively, press Ctrl+a, type :quit and confirm it by pressing Enter.

Serialclient

Serialclient[2] is a CLI client for serial connection written in ruby. Install ruby package, then install it with the following: 

# gem install serialclient

Then, you can use like this: 

$ serialclient -p /dev/ttyS0

tinyserial

tinyserialAUR is a minicom replacement for accessing serial ports on Linux inspired by FreeBSD 'tip'. 

$ com /dev/ttyS0 9600

tio

tioAUR is a simple serial device tool which features a straightforward command-line and configuration file interface to easily connect to serial TTY devices for basic I/O operations. It has less focus on classic terminal/modem features and more focus on the needs of embedded developers and hackers. tio was originally created to replace screen for connecting to serial devices when used in combination with tmux. 

$ tio /dev/ttyUSB0

Graphical front-ends

  • cutecom — A GUI enabled serial monitor.
https://gitlab.com/cutecom/cutecom || cutecomAUR
  • PuTTY — A terminal integrated SSH/Telnet client.
https://www.chiark.greenend.org.uk/~sgtatham/putty/ || putty
  • moserial — A GTK-based serial terminal, primarily intended for technical users and hardware hackers who need to communicate with embedded systems, test equipment, and serial consoles.
https://wiki.gnome.org/moserial || moserial
  • gtkterm — GTKTerm is a simple, graphical serial port terminal emulator for Linux and possibly other POSIX-compliant operating systems. It can be used to communicate with all kinds of devices with a serial interface, such as embedded computers, microcontrollers, modems, GPS receivers, CNC machines and more.
https://github.com/Jeija/gtkterm || gtktermAUR

Windows clients

On Windows machines, connect to the serial port using programs like PuTTY or Terminalbpp.

Installing Arch Linux using the serial console

Note: The Arch Linux monthly release(i.e. the installation medium)'s boot loader has been configured[3] to listen on 0 port(ttyS0/COM1) at 115200 bps, with 8 data bits, no parity bit and 1 stop bit-times.
  1. Connect to the target machine using the method described above.
  2. Boot the target machine using the Arch Linux installation medium.
  3. When the bootloader appears, select Boot Arch Linux (<arch>) and press Tab to edit
  4. Append console=ttyS0,115200 and press Enter.
  5. Now systemd should detect ttyS0 and spawn a serial getty on it. Login as root and start the installation as usual.
Note:
  • After setup is complete, the console settings will not be saved on the target machine; in order to avoid having to connect a keyboard and monitor, configure console access on the target machine before rebooting.
  • While a port speed of 9600 is used in most of the examples in this document, working with higher values is recommended (List of available speeds is displayed in Minicom by pressing Ctrl+a and then p)

Debugging an unresponsive machine using a serial console

Even though [4] has only raw and terse instructions, it presents the full scene. It is important to note that here, the machine under test got unresponsive in a reproducible manner. And that it happened during normal operation. So it could be accessed normally before it needed debugging. However, in general, the serial console is also useful for debugging boot issues. Perhaps by configuring the boot loader by hand at machine startup time. Also note the mentioned netconsole within the P.S paragraph of the external link from this section.

Troubleshooting

Ctrl+c and Minicom

If you are having trouble sending a Ctrl+c command through minicom you need to switch off hardware flow control in the device settings (minicom -s), which then enables the break.

Resizing a terminal

Unlike ssh, serial connections do not have a mechanism to transfer something like SIGWINCH when a terminal is resized. This can cause weird problems with some full-screen programs (e.g. less) when you resize your terminal emulator's window.

Resizing the terminal via stty is a workaround: 

$ stty rows lines cols columns

However, this requires you to manually input the proper geometry. The following methods should be simpler.

1. There is a lesser-known utility called resize, shipped with xterm, that can solve this problem. Invoke it without parameters after you resize the terminal emulator's window: 

$ resize

2. If you do not want to install xterm, it is possible to do the same work via a shell function. Put the following function into your bash/zshrc and invoke it without parameters after resizing the terminal emulator's window: 

rsz() {
	if [[ -t 0 && $# -eq 0 ]];then
		local IFS='[;' escape geometry x y
		echo -ne '\e7\e[r\e[999;999H\e[6n\e8'
		read -t 5 -sd R escape geometry || {
			echo unsupported terminal emulator. >&2
			return 1
		}
		x="${geometry##*;}" y="${geometry%%;*}"
		if [[ ${COLUMNS} -eq "${x}" && ${LINES} -eq "${y}" ]];then
			echo "${TERM} ${x}x${y}"
		elif [[ "$x" -gt 0 && "$y" -gt 0 ]];then
			echo "${COLUMNS}x${LINES} -> ${x}x${y}"
			stty cols ${x} rows ${y}
		else
			echo unsupported terminal emulator. >&2
			return 1
		fi
	else
		echo 'Usage: rsz'
	fi
}

Extra hardware serial ports

The generic 8250 serial driver exposes 32 hardware serial ports, as set in the Arch kernel configuration. This means by default serial ports are created numbered from /dev/ttyS0 to /dev/ttyS31. On most systems many of these ports will be non-functional.

The number can be reduced by setting the kernel parameter 8250.nr_uarts. E.g.: 

8250.nr_uarts=5

This value must be set as a kernel boot parameter not a module option, as the serial8250 driver is compiled into the kernel image.