Christopher Hallinan - Embedded Linux Primer - A Practical, Real-World Approach

• Compilers use optimization techniques that complicate the correlation of C source to actual machine instruction flow. Inline functions are a good example of this.

• Single-stepping through compiler optimized code often produces unusual and unexpected results.

• Virtual memory isolates user space memory from kernel memory and can make various debugging scenarios especially difficult.

• Some code cannot be stepped through with traditional debuggers.

• Startup code can be especially difficult because of its proximity to the hardware and the limited resources available (for example, no console, limited memory mapping, and so on).

The Linux kernel has matured into a very high-performance operating system capable of competing with the best commercial operating systems. Many areas within the kernel do not lend themselves to easy analysis by simply reading the source code. Knowledge of the architecture and detailed design are often necessary to understand the code flow in a particular area. Several good books are available that describe the kernel design in detail. Refer to Section 14.6.1, "Suggestions for Additional Reading," for recommendations.

GCC is an optimizing compiler. By default, the Linux kernel is compiled with the -O2 compiler flag. This enables many optimization algorithms that can change the fundamental structure and order of your code. [89] See the GCC manual referenced at the end of this chapter in Section 14.6.1, "Suggestions for Additional Reading" for details on the optimization levels. For example, the Linux kernel makes heavy use of inline functions. Inline functions are small functions declared with the inline keyword, which results in the function being included directly in the execution thread instead of generating a function call and the associated overhead. [90] Inline functions are like macros, but with the advantage of compile-time type checking. Inline functions require a minimum of -O1 optimization level. Therefore, you cannot turn off optimization, which would be desirable for easier debugging.

In many areas within the Linux kernel, single-stepping through code is difficult or impossible. The most obvious examples are code paths that modify the virtual memory settings. When your application makes a system call that results in entry into the kernel, this results in a change in address space as seen by the process. In fact, any transition that involves a processor exception changes the operational context and can be difficult or impossible to single-step through.

Two popular methods enable symbolic source-level debugging within the Linux kernel:

• Using KGDB as a remote gdb agent

• Using a hardware JTAG probe to control the processor

We cover JTAG debugging in Section 14.4, "Hardware-Assisted Debugging."

KGDB (Kernel GDB) is a set of Linux kernel patches that provide an interface to gdb via its remote serial protocol. KGDB implements a gdb stub that communicates to a cross-gdb running on your host development workstation. Until very recently, KGDB on the target required a serial connection to the development host. Some targets support KGDB connection via Ethernet, although this is relatively new. Complete support for KGDB is still not in the mainline kernel.org kernel. You need to port KGDB to your chosen target or obtain an embedded Linux distribution for your chosen architecture and platform that contains KGDB support. Most embedded Linux distributions available today support KGDB.

Figure 14-1 describes the KGDB debug setup. Up to three connections to the target board are used. Ethernet is used to enable NFS root mount and telnet sessions from the host. If your board has a ramdisk image in Flash that it mounts as a root file system, you can eliminate the Ethernet connection.

Figure 14-1. KGDB debug setup

A serial port is dedicated for the connection between KGBD and gdb running on the development host system, and an optional second serial port serves as a console. Systems that have only one serial port make KGDB somewhat more cumbersome to use.

As you can see in Figure 14-1, the debugger (your cross-version of gdb) runs on your development host system. KGDB is part of the kernel running on your target system. KGDB implements the hooks required to interface gdb with your target board to enable features such as setting breakpoints, examining memory, and enabling single-step program execution.

KGDB is a kernel feature and must be enabled in your kernel. KGDB is selected from the Kernel Hacking menu, as shown in Figure 14-2. As part of the configuration, you must select the serial port for KGDB to use. Notice also from Figure 14-2 that we enabled the option to compile the kernel with debug information. This adds the -g compiler flag to the build process to enable symbolic debugging.

Figure 14-2. Kernel configuration for KGDB

After your kernel is built with KGDB support, it must be enabled. Unfortunately, the method to enable it is not yet uniform across all architectures and implementations. In general, KGDB is enabled by passing a command line switch to the kernel via the kernel command line. If KGDB support is compiled into the kernel but not enabled via a command line switch, it does nothing. When KGDB is enabled, the kernel stops at a KGDB-enabled breakpoint very early in the boot cycle to allow you to connect to the target using gdb. Figure 14-3 shows the logic for generating an initial breakpoint when KGDB is enabled.

Figure 14-3. KGDB logic

KGDB requires a serial port for connection to the host. [91] Notwithstanding the comments made earlier about KGDB over Ethernet. The first step in setting up KGDB is to enable a serial port very early in the boot process. In many architectures, the hardware UART must be mapped into kernel memory before access. After the address range is mapped, the serial port is initialized. Debug trap handlers are installed to allow processor exceptions to trap into the debugger.

Listing 14-1 displays the terminal output when booting with KGDB enabled. This example is based on the AMCC 440EP Evaluation Kit (Yosemite board), which ships with the U-Boot bootloader.

Listing 14-1. Booting with KGDB Enabled Using U-Boot

=> sete bootargs console=ttyS1,115200 root=/dev/nfs rw ip=dhcp gdb

=> bootm 200000

## Booting image at 00200000 ...

Image Name: Linux-2.6.13

Image Type: PowerPC Linux Kernel Image (gzip compressed)

Data Size: 1064790 Bytes = 1 MB

Load Address: 00000000

Entry Point: 00000000

Verifying Checksum ... OK

Uncompressing Kernel Image ... OK

$T0440:c000ae5c;01:c0205fa0;#d9 <<< See text

Most of the boot sequence is familiar from our coverage of U-Boot in Chapter 7, "Bootloaders." This kernel boot sequence has two unique features: the command-line parameter to enable KGDB and the odd-looking text string after the kernel is uncompressed.