Christopher Hallinan - Embedded Linux Primer - A Practical, Real-World Approach

29.80 0.034262 189 181 send

18.46 0.021226 1011 21 10 open

14.11 0.016221 130 125 read

11.87 0.013651 506 27 8 stat64

5.88 0.006762 193 35 select

5.28 0.006072 76 80 fcntl64

3.47 0.003994 65 61 time

2.79 0.003205 3205 1 execve

1.71 0.001970 90 22 3 recv

1.62 0.001868 85 22 close

1.61 0.001856 169 11 shutdown

1.38 0.001586 144 11 accept

0.41 0.000470 94 5 mmap2

0.26 0.000301 100 3 mprotect

0.24 0.000281 94 3 brk

0.17 0.000194 194 1 1 access

0.13 0.000150 150 1 lseek

0.12 0.000141 47 3 uname

0.11 0.000132 132 1 listen

0.11 0.000128 128 1 socket

0.09 0.000105 53 2 fstat64

0.08 0.000097 97 1 munmap

0.06 0.000064 64 1 getcwd

0.05 0.000063 63 1 bind

0.05 0.000054 54 1 setsockopt

0.04 0.000048 48 1 rt_sigaction

0.04 0.000046 46 1 gettimeofday

0.03 0.000038 38 1 getpid

------ ----------- ----------- --------- --------- -----------

100.00 0.114985 624 22 total

This is a very useful way to get a high-level view of where your application is consuming time and where errors are occurring. Some errors might be a normal part of your application's operation, but others might be consuming time that you hadn't intended. From Listing 13-6, we can see that the syscall with the longest duration was the execve(), which is the call that the shell used to spawn the application. As you can see, it was called only once. Another interesting observation is that the send() system call was the most frequently used syscall. This makes sensethe application is a small web server.

Bear in mind that, like the other tools we have been discussing here, strace must be compiled for your target architecture. strace is executed on your target board, not your development host. You must use a version that is compatible with your architecture. If you purchase a commercial embedded Linux distribution, you should make sure that this utility is included for your chosen architecture.

The ltrace and strace utilities are closely related. The ltrace utility does for library calls what strace does for system calls. It is invoked in a similar fashion: Precede the program to be traced by the tracer utility, as follows:

$ ltrace ./example

Listing 13-7 reproduces the output of ltrace on a small example program that executes a handful of standard C library calls.

Listing 13-7. Example ltrace Output

$ ltrace ./example

__libc_start_main(0x8048594, 1, 0xbffff944, 0x80486b4, 0x80486fc

malloc(256) = 0x804a008

getenv("HOME") = "/home/chris"

strncpy(0x804a008, "/home", 5) = 0x804a008

fopen("foo.txt", "w") = 0x804a110

printf("$HOME = %s\n", "/home/chris"$HOME = /home/chris

) = 20

fprintf(0x804a110, "$HOME = %s\n", "/home/chris") = 20

fclose(0x804a110) = 0

remove("foo.txt") = 0

free(0x804a008) =

+++ exited (status 0) +++

$

For each library call, the name of the call is displayed, along with varying portions of the parameters to the call. Similar to strace, the return value of the library call is then displayed. As with strace, this tool can be used on programs for which source code is not available.

As with strace, a variety of switches affect the behavior of ltrace. You can display the value of the program counter at each library call, which can be helpful in understanding your application's program flow. As with strace, you can use -c to accumulate and report count, error, and time statistics, making a useful simple profiling tool. Listing 13-8 displays the results of our simple example program using the -c option.

Listing 13-8. Profiling Using ltrace

$ ltrace -c ./example

$HOME = /home/chris

% time seconds usecs/call calls function

------ ----------- ----------- --------- ----------------

24.16 0.000231 231 1 printf

16.53 0.000158 158 1 fclose

16.00 0.000153 153 1 fopen

13.70 0.000131 131 1 malloc

10.67 0.000102 102 1 remove

9.31 0.000089 89 1 fprintf

3.35 0.000032 32 1 getenv

3.14 0.000030 30 1 free

3.14 0.000030 30 1 strncpy

------ ----------- ----------- --------- ----------------

100.00 0.000956 9 total

The ltrace tool is available only for programs that have been compiled to use dynamically linked shared library objects. This is the usual default, so unless you explicitly specify -static when compiling, you can use ltrace on the resulting binary. Again similar to strace, you must use an ltrace binary that has been compiled for your target architecture. These utilities are run on the target, not the host development system.

With the possible exception of strace and ltrace, no tools are more often neglected by the embedded systems developer than top and ps. Given the myriad options available for each utility, we could easily devote an entire chapter to these useful system-profiling tools. They are almost universally available in embedded Linux distributions.

Both of these utilities make use of the /proc file system, as described in Chapter 9, "File Systems." Much of the information they convey can be learned from the /proc file system if you know what to look for and how to parse the resulting information. These tools present that information in a convenient human-readable form.

The ps utility lists all the running processes on a machine. However, it is very flexible and can be tailored to provide much useful data on the state of a running machine and the processes running on it. For example, ps can display the scheduling policy of each process. This is particularly useful for systems that employ real-time processes.

Without any options, ps displays all processes with the same user ID as the user who invoked the command, and only those processes associated with the terminal on which the command was issued. This is useful when many jobs have been spawned by that user and terminal.

Passing options to ps can be confusing because ps supports a wide variety of standards (as in POSIX versus UNIX) and three distinct options styles: BSD, UNIX, and GNU. In general, BSD options are single or multiple letters, with no dash. UNIX options are the familiar dash-letter combinations, and GNU uses long argument formats preceded by double dashes. Refer to the man page for details of your ps implementation.

Everyone who uses ps likely has a favorite invocation. One particularly useful general-purpose invocation is ps aux. This displays every process on the system. Listing 13-9 is an example from a running embedded target board.

Listing 13-9. Process Listing

$ ps aux

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

root 1 0.0 0.8 1416 508 ? S 00:00 0:00 init [3]

root 2 0.0 0.0 0 0 ? S< 00:00 0:00 [ksoftirqd/0]

root 3 0.0 0.0 0 0 ? S< 00:00 0:00 [desched/0]