Christopher Hallinan - Embedded Linux Primer - A Practical, Real-World Approach

Listing 14-24. Trapping Crash on Panic Using KGDB

$ ppc-_4xx-gdb --silent vmlinux

(gdb) target remote /dev/ttyS0

Remote debugging using /dev/ttyS0

Malformed response to offset query, qOffsets

(gdb) target remote /dev/ttyS0

Remote debugging using /dev/ttyS0

breakinst () at arch/ppc/kernel/ppc-stub.c:825

825 }

(gdb) c

Continuing.

<< KGDB gains control from panic() on crash >>

Program received signal SIGSEGV, Segmentation fault.

0xc0215d6c in pcibios_init () at arch/ppc/kernel/pci.c:1263

1263 *(int *)-1 = 0;

(gdb) bt

#0 0xc0215d6c in pcibios_init () at arch/ppc/kernel/pci.c:1263

#1 0xc020e728 in do_initcalls () at init/main.c:563

#2 0xc020e7c4 in do_basic_setup () at init/main.c:605

#3 0xc0001374 in init (unused=0x20) at init/main.c:677

#4 0xc00049d0 in kernel_thread ()

Previous frame inner to this frame (corrupt stack?)

(gdb)

The crash in this example was contrived by a simple write to an invalid memory location (all ones). We first establish a connection from gdb to KGDB and allow the kernel to continue to boot. Notice that we didn't even bother to set breakpoints. When the crash occurs, we see the line of offending code and get a nice backtrace to help us determine its cause.

• Linux kernel debugging presents many complexities, especially in a cross-development environment. Understanding how to navigate these complexities is the key to successful kernel debugging.

• KGDB is a very useful kernel-level gdb stub that enables direct symbolic source-level debugging inside the Linux kernel and device drivers. It uses the gdb remote protocol to communicate to your host-based cross-gdb.

• Understanding (and minimizing) compiler optimizations helps make sense of seemingly strange debugger behavior when stepping through compiler-optimized code.

• gdb supports user-defined commands, which can be very useful for automating tedious debugging tasks such as iterating kernel linked lists and accessing complex variables.

• Kernel-loadable modules present their own challenges to source-level debugging. The module's initialization routine can be debugged by placing a breakpoint in module.c at the call to module->init().

• printk and the Magic SysReq key provide additional tools to help isolate problems during kernel development and debugging.

• Hardware-assisted debugging via a JTAG probe enables debugging Flash or ROM resident code where other debugging methods can be cumbersome or otherwise impossible.

• Enabling CONFIG_SERIAL_TEXT_DEBUG on architectures where this feature is supported is a powerful tool for debugging a new kernel port.

• Examining the printk log_buf often leads to the cause of a silent kernel crash on boot.

• KGDB passes control to gdb on a kernel panic, enabling you to examine a backtrace and isolate the cause of the kernel panic.

Linux Kernel Development , 2nd Edition

Robert Love

Novell Press, 2005

The Linux Kernel Primer

Claudia Salzberg Rodriguez et al.

Prentice Hall, 2005

"Using the GNU Compiler Collection"

Richard M. Stallman and the GCC Developer Community GNU Press, a division of Free Software Foundation

http://gcc.gnu.org/onlinedocs/

KGDB Sourceforge home page

http://sourceforge.net/projects/KGDB

Debugging with GDB

Richard Stallman, Roland Pesch, Stan Shebs, et al.

Free Software Foundation

www.gnu.org/software/gdb/documentation/

Tool Interface Standards

DWARF Debugging Information Format Specification

Version 2.0

TIS Committee, May 1995

In the previous chapter, we explored the use of GDB for debugging kernel code and code resident in Flash, such as bootloader code. In this chapter, we continue our coverage of GDB for debugging application code in user space. We extend our coverage of remote debugging and the tools and techniques used for this peculiar debugging environment.

We already explored several important debugging tools in Chapter 13, "Development Tools." strace and ltrace can be used to observe and characterize a process's behavior and often isolate problems. dmalloc can help isolate memory leaks and profile memory usage. ps and top are both useful for examining the state of processes. These relatively small tools are designed to run directly on the target hardware.

Debugging Linux application code on an embedded system has its own unique challenges. Resources on your embedded target are often limited. RAM and nonvolatile storage limitations might prevent you from running target-based development tools. You might not have an Ethernet port or other high-speed connection. Your target embedded system might not have a graphical display, keyboard, or mouse.

This is where your cross-development tools and an NFS root mount environment can yield large dividends. Many tools, especially GDB, have been architected to execute on your development host while actually debugging code on a remote target. GDB can be used to interactively debug your target code or to perform a postmortem analysis of a core file generated by an application crash. We covered the details of application core dump analysis in Chapter 13.

Cross-development tools were developed primarily to overcome the resource limitations of embedded platforms. A modest-size application compiled with symbolic debug information can easily exceed several megabytes. With cross-debugging, the heavy lifting can be done on your development host. When you invoke your cross-version of GDB on your development host, you pass it an ELF file compiled with symbolic debug information. On your target, there is no reason you can't strip [100] Remember to use your cross-version of strip, for example ppc_82xx-strip. the ELF file of all unnecessary debugging info to keep the resulting image to its minimum size.

We introduced the readelf utility in Chapter 13. In Chapter 14, "Kernel Debugging Techniques," we used it to examine the debug information in an ELF file compiled with symbolic debugging information. Listing 15-1 contains the output of readelf for a relatively small web server application compiled for the ARM architecture.

Listing 15-1. ELF File Debug Info for Example Program

$ xscale_be-readelf -S websdemo

There are 39 section headers, starting at offset 0x3dfd0:

Section Headers:

[Nr] Name Type Addr Off Size ES Flg Lk Inf Al

[ 0] NULL 00000000 000000 000000 00 0 0 0

[ 1] .interp PROGBITS 00008154 000154 000013 00 A 0 0 1

[ 2] .note.ABI-tag NOTE 00008168 000168 000020 00 A 0 0 4

[ 3] .note.numapolicy NOTE 00008188 000188 000074 00 A 0 0 4

[ 4] .hash HASH 000081fc 0001fc 00022c 04 A 5 0 4

[ 5] .dynsym DYNSYM 00008428 000428 000460 10 A 6 1 4

[ 6] .dynstr STRTAB 00008888 000888 000211 00 A 0 0 1

[ 7] .gnu.version VERSYM 00008a9a 000a9a 00008c 02 A 5 0 2