Using the Debuggers

This is largely useful for when the source code of the file is unavailable.

This guide is aiming to help people who wish to apply an understanding of the Assembly-level instructions that computer hardware will use.

There are some other things, like some basic information gathering, that may be able to be done with some of the same software, but without requiring the contemplation of Assembly code. Compared to fully analyzing the Assembly code, such basic information-gathering techniques may be less likely to provide as complete of an understanding of precisely what the hardware does. On the other hand, such diagnostic techniques may be much easier, and may be much faster, than interpreting each individual op code. The section on crash handling provides some details about such activities.

Listing some available software

Examples might include:

Some programs commonly used in Unix
  • gdb
  • trace? strace?
  • objdump
  • nm (shows information related to “names”. Such “names” are more commonly referred to as “symbols”.)
  • strings (for non-obscucated)

A lot of that software may be on platforms such as operating systems that use Unix, as well as CygWin.

DOS DEBUG

DOS also came with a debug.exe which is not covered extensively here.

Debugging in Microsoft Windows

Besides possible Unix ports (like Cygwin), or maybe DOS programs (though such software may be unlikely to work well in modern Microsoft Windows systems, especially when using 64-bit operating systems), Microsoft has released “Debugging Software for Windows”.

This section currently does not have further details on using “Debugging Software for Windows” for debugging code. However, details about getting some software can currently be found on the crash handling page.

Wikipdia's list to compare debuggers, Wikipedia's article on Debuggers: section called “List of debuggers”. Norman Matloff's Data Display Debugger (DDD) Tutorial Center (first hyperlink is to actual tutorial, which is a PDF file that appears to be the output of an old slideshow or web page).

Many people like being able to use a graphical interface, where setting a breakpoint may be (nearly) as easy as clicking on a line of code.

An opinion

Learning the text mode debuggers first may be worthwhile, because they aren't too terribly hard to learn. The text mode debuggers may also have some more advanced/detailed/deep options, which could eventually provide more flexibility. After learning the basics with a text editor, people who need more functionality can investigate more advanced options. People who feel like they have obtained everything they wanted, but who still want to look for an interface that may be more pleasant, could look into front-ends at that time. If one front-end doesn't feel nice, switching to another front-end may be more straightfoward when the basics were already learned from a text mode debugger.

For further information about available front-ends: Wikipedia's article on Debuggers: section called “Debugger front-ends”. Also, Wikipedia's article called “Comparison of debuggers” has a section called “List of debuggers”.

Some possible recommendations, based largely on very little research: LLVM's debugger, GDB, Xcode, Eclipse (IDE)

Know what you're getting into
Overview

There are various file formats of executables files. An executable file format is namely designed around support for a specific CPU/“assembly language” and a specific “operating system” environment.

Many of the formats contain a reference to an operating system that supports the format. This doesn't necessarily mean that those are the only operating systems to support the format. For example, i086-dos16.exe (“DOS 16-bit executable”) are likely to be supported by 16-bit versions of DOS (MS-DOS and compatible), but may also be supported by some versions of 32-bit (or 16-bit) Microsoft Windows environments.

Surely there are ways of storing executable code other than just what is supported by the UPX software. For instance, code for Amiga computers or 6502 CPUs (used by the 8-bit Nintendo Entertainment System and the Commodore 64) may have used different formats.

The home page for the “Universal Packer for eXecutables (“UPX”)” software lists several different executable file formats supported by that software. Wikipedia's comparison of executable file formats mentions some other formats.

The “Universal Packer for eXecutables” used the word “universal” in its name because the software supported several different executable formats, although the software mainly supports file formats that are both designed for executable code, and which have also been in some widespread use (since the UPX software has been created). Many of the formats in that list will mention being either “ELF” or “PE”. Here is a brief look at some relatively common file foramts:

ELF (“Executable and Linker Format”)
Uses an “ELF” file format
PE (“Portable Executable”)
Uses a “PE” file format. This format is used by Microsoft Windows (32-bit versions, and newer). PE stands for “Portable Executable”
a.out

Prior to widespread use of some newer file formats (like ELF or COFF), executable files in some operating systems (including old Unix operating systems, and also including Linux) were typically using a format called “a.out”. For this reason, the GCC software has a default output filename of “a.out”. Even after GCC started supporting the ELF file format, GCC continued to use the same default filename.

Universal/Fat binaries

A “file format” refers to the rules that were followed when the file was created. It is possible that a single executable file may be able to fulfill two sets of rules, and so a single file may actually be a legitimate, properly formatted file in two different file formats. The general concept is discussed in further detail by Wikipedia's page about “Fat binary”. That web page mentions a very old example of a single executable file designed for both CP/M-80 and MS-DOS, by using bytes 0xEB52EB (a.k.a. “EBh, 52h, EBh”). Although that specific example is quite dated, the details do describe how this works.

In addition to the term “Fat binary”, such files are sometimes called “universal” binary executable files.

The concept of successfully creating a “Fat binary” can be technicailly challenging or even impossible (if two standards have mutually imcompatible rules), and often provides very little benefit. Instead of two seperate small files, one big file may contain code designed for two different platforms. This means less files to deal with, which is nicer, but having smaller files can also be nicer. Such files have typically not been used as widely as making files designed for a single way of being executed. Still, because such files exist, it may be nice to know about their existance (especially in case such a file gets mis-detected by some sort of simple automatic format-detection process).

What need to be done

Before trying to debug an executable file, know which file format is being used by the executable file. (In some cases, knowing the exact file format might not be an absolute requirement, but having some idea is generally a good first step.)

Often, the list of formats may be narrowed down significantly by knowing what operating system is (or will be) used to run the executable file. For example, a 64-bit AMD64-compatible operating system that uses the Linux kernel is likely to use the AMD64-Linux.ELF file format, and not the i086-dos16.sys (16-bit DOS *.sys file) format. Actually, one might not be 100% certain that a specific file type is not being used, due to compatibility (which might be provided using emulation/virtualization).

Starting, Opening file

If the debugger is built into an “integrated development environment”, then the debugger may be able to find the code by looking at what is open in the “integrated development environment” software. In this way, the debugger may even be able to see source code that hasn't been saved to a user-specified filename.

In other cases, specifying the name of the program may be needed.

GDB: Opening a file, interface
gdb exefile

The filename is required, but specifying it on the command line is only the recommended approach, for simlicity and speed (as there is slightly less typing). The other option is to simply run the assembler (by running gdb and then specifying the executable's filename within in the gdb program.

(gdb) file exefile

When a file's name is specified, then gdb may report the specified name, and show “: Success.” However, simply seeing that success message does not indicate that the program will be working in the way that is probably desired. If that is the only success message shown, the program will probably not be working as desired. If the executable file has symbols, the recommended way to identify success is to also look for confirmation that the symbols have been loaded. (If this is done automatically from the command line, then this may be the last line of initial output shown before the first prompt.) Example output is:

Reading symbols from exefile...done.
Support by Emacs

Note: There are various versions/derivatives of Emacs. It is suspected that the techniques described here may not work with all variations.

GDB may also work particularly well with emacs. People who have used plug-ins in programs may initially think that it seems silly to try to use a text editor as a main front-end for a complex program like a debugger. However, Richard Matthew Stallman is behind the creation of both GDB and emacs, so such support isn't too terribly surprising. Also, the truth is, Emacs provides a windowing system that may be nicer than trying to use GDB directly. So, let's see how to do this a bit.

Some Emacs basics

First of all, if you start to use EMacs, you may need to know about the “Meta key”.

Starting Emacs

If using GNU EMacs in a graphical environment, but you wish to not use a graphical interface, start it with the --no-window-system parameter, as shown in this example (which uses the abbreviated version of the parameter).

emacs -nw

Emacs hasn't even started making itself useful yet, but since it can be nice to know how to properly exit a program, see the section on closing Emacs to learn this essential detail about working with this program.

If the backspace key does not work as expected, terminal settings are not providing an ideal experience. One option may be to try to adjust terminal settings; a far quicker option may be to just try pressing ^H (Ctrl-H) as that might produce desired results.

(When specifying Ctrl key combinations with uppercase letters, lowercase letters work just as well. This is noted by control keyboard sequences.)

Okay, enough about general Emacs processes like quitting. Let's show how this program integrates with GDB. Run Emacs if it is not running yet.

Press Meta-x. (Quick reminder: anyone unfamiliar with the “Meta key” is recommended to become familiar with the “Meta key”).

Emacs should show an “M-X ” prompt. At that prompt, type:

gdb

Note: Emacs documentation on GDB indicates that using this command limits the user to debugging a single executable file (at one time).

After the user presses Enter, Emacs may prompt the user, with something like this:

Run gdb (like this): gdb --annotate=3 exefile

The “ --annotate=3” is probably fine, and so may be ignored. Make sure that the specified filename (which is what comes after the “ --annotate=3”) is specifying the file that should be getting debugged. (If not, then fix the problem.) Then, press Enter.

Loading symbols

Check that the symbols were successfully read (if symbols are available). Some versions of GDB will display this when the file is started, by stating “Reading symbols from exefile...done.”. Other versions of GDB have been known to not show this line initially. If debugging symbols are not found, GDB will typically output, “(no debugging symbols found)”. If there is doubt, debugging symbols may be re-loaded by using gdb's internal “file” command, specifying the executable filename (e.g.: “file exefile”) from within GDB. (This is different than the file command at a system command prompt.) If debugging symbols are avaialble, GDB may say, “Load new symbol table from "/full/path/to/exefile"? (y or n)”. Then, say “y” (for “yes”). (There is no harm caused by doing this if the same symbols were loaded earlier. So, if there is doubt on whether this is needed, doing this multiple times is not expected to be a problem.)

With GDB 6.3 (found in an OpenBSD package), running this file command from within GDB will show “Reading symbols from exefile...done.”, even if that message doesn't show up automatically when GDB is started. So, if you don't see the message on startup, manually re-loading the symbols may show a satisfying message.

Loading up the Emacs-enhanced interface in GDB

Now that GDB is already running in Emacs, it is time to place GDB/Emacs into a windowed mode.

Press Meta-x again, and type:

gdb-ma

Then press Tab a couple of times, and the command will likely complete for you. The full command to be typed is:

gdb-many-windows

After typing that command, press Enter.

Suddenly, gdb is in a windows mode. Each individual window may be pretty small on an 80x24 (columns x rows) display. Using a larger display is recommended. (If the terminal window is resized, even remotely over SSH, then Emacs may be smart enough to auto-resize.)

Note that pressing Esc three times may cause Emacs to get out of the special gdb windowed mode. Chances may be quite high that this is undesirable (because the window that remains visible may not be the desired one). So, you've been warned.

Working with the Emacs windows
Using built-in commands

Pressing ^x (that is, Ctrl-X) and then o will move from window to window. (This is less important to know if a working rodent pointer is available.)

To jump back a window, the simplest built-in keystroke sequence may be: ^U -1 ^X o (without spaces... that is: Ctrl-U, then type -1 (that's a dash and then the number one), then type Ctrl-X, then type lowercase o.

That can be thought of as a mixture of two commands. The Ctrl-U specifies the number of times that a repeatable command is being requested. The -1 specifies how many times the repeatable command should be performed. Finally, the Ctrl-X and then “o” is set up to use the “other-window” command. (This can be verified by viewing the keyboard bindings, which may be viewed by looking at the related help content that is built into Emacs. See viewing Emacs keyboard bindings for more details.)

The “other-window” command receives the details that “-1” is the requested number of times to perform the action. The “other-window” command happens to understand the idea of using a negative number (which causes the cycle through windows in the opposite order as when a positive number is requested.

Before moving on to other topics, take just a moment to realize this feature of Emacs. Pressing ^U and then specifying a number can cause many commands to be repeated. (This can even be used for the command of simply pressing a character that should go into a text file. Need 16 spaces? Just press ^U and then 16 and then press the space bar.

Other options

Some places have noted that pressing ^-- (that is: control-hyphen, a.k.a. control-“minus sign”) may be used instead of ^U -1. Preliminary testing indicated that this does not work, but since it has been mentioned multiple times, perhaps this shortcut works better on some systems than others. (Perhaps the numpad's “minus sign” works better than the main keyboard's minus sign?)

....................................................................................................................... at looks pretty complicated, and there are some other options. However, discussing other options further, there may be some benefit to analyzing that particular sequence. The Ctrl-U command means to perform a repeatable sequence a certain number of times. The -1 specifies how many times: in this case, negative one. A different value (like 3) could have been specified to jump to a different window. Then, Ctrl-X followed by o runs the “other-window” command. What the “other-window” command does is to check how many times it is being asked to repeat. (The default number of times may be one, meaning that the “other-window” command will perform its action one time. If that “other-window” command is asked to perform its action a positive number of times, then the “other-window” command runs the “next-window” command (however many times the “other-window” command is supposed to perform its action). However, if the “other-window” command is being given a negative number, then “other-window” command runs the “prev-window” command.

One of the key items to take away from that point is that Emacs does support a repeat command. Many other commands may be repeated. Specifying a negative number might not make any sense for some commands. However, when using the “other-window” command, there is a logical action to perform when a negative number is specified. (The logical action is to move the cursor in the opposite direction as what happens when a positive number is specified.) Some of the details about the behavior of the “other-window” command are documented by the inline help, which can be seen by viewing the ^X keyboard bindings. (If “^X o” is performing its default action, then the “keyboard bindings” help page will have a hyperlink to the help page for the “other-window” command). Describing the comamnds .........................................................................................................................................................................................................................................

There may be other ways to move between windows with the keyboard. For instance, one unpleasantly long way is to start by pressing M-x, and then typing “next-multiframe-window” or “previous-multiframe-window” (which may be abbreviated down to “prev-mu”), and then pressing Enter. That seems even more complicated, but the benefit is that hotkeys can be created to make that simpler (allowing an easy way to move forward, or backwards). For instance, StackOverflow page suggests making hotkeys. If setting up the hotkeys is something that is regularly desirable, then the commands to set up those hotkeys can be placed in a configuration file that Emacs loads. Other solutions may be mentioned by Emacs Wiki: switching between windows (which may show how to use a hotkey sequence that involves using the Shift key?), or StackOverflow page on moving between Emacs windows.

As an example of what may be done in this mode, pressing Ctrl-X and then the Space bar may insert a breakpoint. (If there is already a “breakpoint bar” shown to the left of the code, which might be typical in a graphical environment, then this also may be doable by using a mouse, clicking to the left of the rest of the column of text.) The breakpoint may be visible with a red symbol in the left column. (If Emacs is being displayed in text mode, this symbol may be a bold letter B. In a more graphical environment, Emacs may show another symbol (an octagon? A circle?). Also, the lower-right window may default to showing breakpoints.

For additional commands, press ^H (that is, Ctrl-H followed by b to temporarily overwrite a sub-window with help that shows available keyboard bindings. Unfortunately, the sub-window that gets used may be a different one than intended. Use ^x (that is, Ctrl-x) o to move the cursor to that sub-window, and then press q to quit the help. Another bit of help is available by pressing ^H (that is: Ctrl-H) followed by either Ctrl-H again, or followed by just h. For even more help, press Ctrl-H followed by some of the other available options that are shown on the main help screen. On these help screens, the most important key to know about may be q to quit the current help screen. (It is possible to be nested into multiple help screens, and pressing q just goes out one layer.) Another keyboard sequence that may be nice to know is Ctrl-S to search. To understand the effect of a particular key or keystroke sequence, Ctrl-H then k and then the keystroke sequence.

This interface may be subject to change: EmacsWiki on GdbMode states, “GDB developers plan to make the annotation interface obsolete.” Additional help may be seen at Emacs documentation related to GDB Graphical Interface.

Other than knowing about q to quit help, and details on how to close Emacs, there may be some other prompts where the user is not sure what to do. In such cases, consider trying the “tab” key (to attempt auto-complete). Or, if that is not providing any satisfaction, try pressing Ctrl-G (which is the “interrupt” character in Emacs) to back out of a prompt.

Displaying registers

To see registers, Emacs manual: Other GDB buffers states, “To switch from one to the other, click”. Other tasks have instructions involving the keyboard. There may be a solution at Emacs GDB local/registers window. Or, as a built-in solution, try M-X followed by gdb-display-registers-buffer.

Here are some scratch notes: This process is still being reviewed/explored:

It appears the upper-right window may be a bit special, as it is “dedicated”. Using a mouse can click on the “Registers” button; there is no real easy way to do that with the keyboard. Checking the keyboard bindings (^H b) will show that the mouse clicks are bound to functinos like mouse-select-window but that “must be bound to an event with parameters”. So using that command with a keybaord might not be the easiest thing. Also, http://markshroyer.com/2012/11/emacs-gdb-keyboard-navigation/ notes: “in the case of the Locals/Registers and Breakpoints/Threads windows, the buffer you want to visit doesn’t necessarily even exist until you click that button.”

There may not be an easy way around this except to muck with the *.el file. Fortunately, Emacs GDB local/registers window provides some code to put in a custom file.

Emacs: dedicated windows


facemenu-menu
mouse-buffer-menu
<S-down-mouse-1>                mouse-appearance-menu
mouse-select-window

mouse-select-window

emacs20 emacs21#sec114 If gdb-display-registers-buffer appears in an undesired window, M-X buffer-menu? M-X, then command, like ones shown: same StackOverflow page ^X ^B shows buffers To change what is shown in a window,k to a different buffer: M-X buffer-menu then put the cursor on a desired line, and press Enter. buffer-list? M-X ^U 3 ^X o : press ^X 3 times ^X z : repeat last command ^X zzzz : repeat last 4 commands ^X u or ^/ or ^_ : undo Emacs Undo Emacs change window Emacs: Displaying buffers Emacs: switching buffers change a window to a desired buffer: ^X b This is the switch-to-buffer command Then, select buffer. You may press tab to show the buffers in one of the windows, or use up arrow/down arrow. switch-to-buffer-preserve-window-point Emacs: switching buffers Emacs-fu blog about GDB including quotes by Dijkstra Emacs-fu color theming Linux Journal article Emacs mdoe: discusses the *.el files At the top of the screen, is the menu bar. menu-bar-open may be done with M-x menu-bar-open or M-` (esc, `) or F10. gdb-frame-registers-buffer gdb-frame-locals-buffer Other GDB buffers "When gdb-many-windows is non-nil, the locals buffer shares its window with the registers buffer, just like breakpoints and threads buffers. To switch from one to the other, click with Mouse-1 on the relevant button in the header line." other keys for older? Other keys for older? info frame info args / info locals = these require symbols Selection Emacs Wiki: GDB-MI Linux Journal article mentions gud.el, gdba.el

Some additional help

Here are some handy commands/keystrokes, including some that provide handy help that is built into the Emacs program:

^G ^G (Ctrl-G once, or twice)
Interrupt
Window manipulation
Closing a window
In a graphical environment, the easiest way may be to click on a button that is visible in a window's border. When using just the keyboard, the easiest way may be to first place the keyboard cursor in the window that should be closed, and then press Ctrl-zero (^0).
Splitting a window

This is how windows get created (as noted by Emacs documentation: Multiple windows).

^X 2 makes a horizontal split, causing two smaller windows where one window will be above the other window. The contents of the window will be the same (approximately, with one of the windows possibly being one line larger than the other).

^X 3 does a similar thing, but uses a vertical split.

Changing window size

^-} tries to horizontally expand a window (which makes more sense on windows that are on the left side of the screen... The right curly brace will try to move the border to the right. For windows on the right side of the screen, this results in the side border being moved left.) ^-{ will do the opposite, trying to horizonally shrink a window.

^x ^ (Ctrl-x, then carrot) will try to vertically grow a window.

To shrink the window by a line, use: ^U -1 ^x ^

^x + (Ctrl-x, then +) will try to size the windows approximately the same. All windows will still be rectangular, and screen space will be used up. Depending on how many windows there are, some windows may be half of the height as other windows.

Changing what is seen in a window/frame

Emacs shows a “buffer” in each window. When people try to use Emacs to modify the contents of a file, what really happens is that Emacs copies the contents of the file into a “buffer”, and then people edit the buffer, and then the buffer is saved to a file.

Creating a new buffer

To create a new buffer from stratch, one way is to use: ^X b (Ctrl-X b). Then, Emacs will prompt for a buffer name. If the user types in a name that is not currently being used, then Emacs will create a new buffer and that buffer will specify the name that the user specified.

Listing buffers

To list available buffers to switch to (that is, all buffers except for the current one), use ^x b, and then press tab or question mark. (Then press the other key, question mark or tab, to uselessly widen the display once.)

Alternatively, ^X ^B (Ctrl-X, Ctrl-B) will show information about existing buffers. This has the advantage of showing filenames that are related to a buffer. An asterisk in the column with the M header indicates that the buffer has been modified.

Switching between buffers

To make an existing window/frame show the contents of a different, existing buffer, “^X b” may be used. Another option may be ^X followed by either left arrow, or right arrow.

^H

Shows basic help. (This can also be done by pressing Meta-x, and then typing “help”, and then pressing Enter.) This may cause a new window to appear. Emacs will be showing a prompt. Scrolling down in the help window may be done by pressing the space bar (or the delete key). Since the keyboard cursor is in the bottom of the screen, standard window-movement keyboard commands are not available. The program will expect to be provided with a topic to provide further help. If this becomes undesirable, press ^G to Interrupt the help command.

^H ^H

This shows help related to help.

This is assuming that the backspace key (represented by Ctrl-H) has been left to its default value of showing help. If the backspace was customized to be deleting characters, then this same help may be reached by pressing Meta-x and then typing help-for-help and pressing Enter.

[#emacshbn]: ^H b

Shows key bindings. (See also: Online help for Emacs key bindings.)

Simply using ^H b is assuming that the backspace key (represented by Ctrl-H) has been left to its default value of showing help. If the backspace was customized to be deleting characters, then this same help may be reached by pressing Meta-x and then typing help and pressing Enter. Then, Emacs is at a prompt asking what help topic to look at. The help that is shown will reveal that the “b” command will show the keyboard bindings. So, go ahead and press the b key, and then press the Enter key.

Once the keyboard bindings are shown, make sure the keyboard is in the window that is displaying the keyboard bindings. Then, pressing / will bring up a prompt to be able to search through that help.

Showing more available keys

Access keyboard help by pressing a prefix key, and then the help key. For instance, Ctrl-X and then Ctrl-H will show help related to the keys that can be pressed after Ctrl-X.

If that doesn't work (because ^H is not the help key, for instance), another attemptable approach is to use another help key (like F1), or just look at the general keyboard bindings page (^x help b) and search (/) for C-x.

Note that the general keyboard bindings may lump many, but not all, of the ^X commands near each other. On at least one system, there were nine different sections that listed one or more commands that started with ^X. (After C-X was C-x 8, and then C-x 8 SPC, and then C-x (again), and then C-x C-@, and then C-x C-a C-b, and then C-x a i g, and then C-x (yet again), and then C-X @.) So, all of the commands ^X-related commands are not grouped together quite as nicely, but they are showing.

Note that there may be additional commands, possibly due to the use of keyboard sequences. Using Ctrl-X Ctrl-H did show help related to Ctrl-X, but that keystroke sequence did not show up in the list of help related to keyboard sequences that use Ctrl-X.

M-x Tab
(Meta-x, then tab): starts by getting to a Meta-x prompt, and then tab completes, showing commands that are available.
Checking what a sequence does

The simplest way to check what Emacs command is performed by a specific keystroke might be: ^H k. After that, type in a command sequence to learn what Emacs will do in response. For example, “^H k ^H k” will end up pulling up the help for the “describe-key” command. This may be imperfect: ^X ^H may show “C-x C-h is undefined” even if, in reality, that sequence pulls up a help menu.

Quitting the Debugger

Sometimes, the debugging environment may get in the way. For instance, in an integrated development environment, the debugging functionality may put some information on the screen. Once the hardcore (detailed) debugging is no longer necessary, the user may wish to make the debugging stuff leave the screen, in order to provide more space on the screen for something else.

GDB

Type q.

If a program is running, then the user will be prompted.

(gdb) q
A debugging session is active.

        Inferior 1 [process 1553] will be killed.

Quit anyway? (y or n) y

If you are using a front-end, that may need to be closed separately. If Emacs is the front-end, closing Emacs may discuss closing that front end.

Inline help
Overview

Some software may provide help from within the program.

GDB
gdb --help

That shows some brief options that may be specified from the command line. However, there is much more help available from within the program.

gdb has a “help” and an “apropos” comamnd. The “help” command may be provided with a word to show more details.

gdb has a “help” and an “apropos” comamnd. The “help” command may be provided with a word to show more details.

The contents of “help all” is a bit lengthy. (Be prepared to press the Enter key several times to scroll through all of the help.) Furthermore, those contents do not seem to be easily available online from the official site. (Even though GDB is open source, the contents of the “help all” screen is actually created by combining information from multiple different source code files. So, to see that information online, third party resources may be best: GDB-Help is one example.

There is a lot of additional help available within the program. For instance, try typing:

(gdb) help x

That shows help for the “examine” command, which is also abbreviated as the “x” command. So, there is a lot of help available as well.

Inline tab-based help is also available:

Using tab help
(gdb) help info 

(That is: the word “help”, a space, the word “info”.)

Then, before pressing Enter, type another space after the word “info”, and press the tab key twice. Many options will show up. Press r and then press the tab key twice. The number of available options will be reduced. (This guide is based on a version of GDB that had only two options that start with r: record, and registers.) All of the options start with “help info re”, and so the “e” is now part of the incomplete command. The GDB program is patiently waiting for clarification on the rest of the command.

Go ahead and type “g”, and press tab twice. That should show the completion of a word.

Additional documentation may be at GDB documentation, including a reference to the official guide, Debugging with GDB (including GDB Command, Variable, and Function Index) which is all made by the Free Software Foundation, and is official documentation.

Unsurprisingly, there are multiple third party sites with information. Examples of such documents include this guide from ][CyberPillar][, and GDB commands by function - simple guide.

Listing Instructions
GDB: Listing Instructions

The nicest way, quite easy to describe, may be to see the lines of source code in the C programing language. If the C source code is available, then listing the instructions will reference the source code, and show some lines related to the source code. For a very small program, the l command might even show the entire program. Otherwise, “l optionalLineNumber” will show multiple lines of sourse code, but not all of it.

Seeing assembly instructions

The assembly instructions may also be shown. To do this, use the “disassamble” command. This may show the disassembled code for the current program. For a program that has been started, but is currently paused, simply type that command to see the current function. Then, look for the line that starts with “=> ”. That points to the instruction that is about to be executed (when the program is resumed); that instruction has not been executed yet.

The instructions that are output will be in AT&T syntax, which GDB calls the “att” flavor. If disassembly into Intel syntax is desired, see:

show disassembly-flavor
set disassembly-flavor intel

Actually, an even more informative option to use “disas /r”. This will show the bytes to the left of the assembly code, making it much more similar to the output that is obtained by running “objdump -d” from a command line.

Another option, to see a list of 5 instructions starting at the one pointed to by the $eip variable:

disas /r $eip $eip+5

disas $eip $eip+5

(The phrase “$eip” refers to the EIP register, which points at the current instruction when using IA-32 x86 assembly. Other variations of assembly may need to use a different named reference.)

It is also possible to try specifying code by specifying a function name. That may or may not work well, depending on what symbols are available.

The following is another way to see some instructions. This shows 5 instructions.

x /5i $eip

objdump
objdump -d exename
This may be able to be narrowed down. e.g.: objdump -d -j .text exename
Program flow

This guide starts by demonstrating how to pause the program at various locations. After all, looking at the contents of a variable at the very start of a program is typically uninteresting. The usefulness of the debugging software is to be able to easily check the state of the (main, debugged) program after the program has been running.

Running software

The debugging software may find a reason to pause the (main, debugged) program that is being run. For instance, if a breakpoint gets set up, the debugging software will stop when the break point is reached.

If the debugging software does not pause the running program, the results may be simply be that the program runs normally, indistinguishable from when the (main, getting-debugged) program runs outside of the debugger software. That is fine. In this case, the debugging software may not be providing substantial benefit. Nor is there likely to be substantial harm from running the (main, debugged) program from within the debugging software.

So, if there is no significant impact from using the debugging software, then why is it being used? Because there are other things that may be done, such as “single stepping”, or pausing because a “breakpoint” may be reached. Those techniques can result in a substantially slower experience, which may be desired when a programmer wants to slow things down and closely inspect what is going on.

It is also nice, however, to be able to run the (main, getting-debugged) program at full speed (or much closer to full speed), quickly passing through code that is expected to be working fine. This section notes how to start software, and how to unpause software. In some cases, those tasks (starting the software from scratch, and unpausing the software) may be done in the same way.

Running software in gdb
Starting the software from the beginning

r optional-command-line-parameters

the “r” command may support using a less than sign to indicate redirection from a text file. e.g.:

r optional-command-line-parameters <input.txt

The r command will cause gdb to check if the file(s) have been updated. If so, it will re-load the new files. (Therefore, quitting gdb and re-running gdb is more work than what is generally needed to run updated files.)

gdb will then run the files.

Resuming after being paused
continue
Pausing the program and reporting values
Breakpoints

Setting a breakpoint (commonly) involves specifying a line of code where the program should be paused. After a breakpoint is set up in the debugging software, when the (main, getting-debugged) program does start running, the program will be automatically paused when the breakpoint is reached.

Where breakpoints may be

To keep things simple, expect the breakpoint to stop at the beginning of a line which is the start of a statement.

Commonly, a breakpoint is set to the start of a line of code in a program, and is also at the start of a statement. So, if there are multiple expressions (like multiplying and adding, as in: “a = (a * b) + (c * d)”), the breakpoint will typically stop before any of that expression starts to be evaluated.

Some programming langauges allow a statement to be spread out over multiple lines. Some programming languages (including languages just described in the previous sentence) allow multiple statements to be started on a single line. Debugging software typically makes a breakpoint effective at the start of a line. So, if there are multiple statements/commands on one line, there may not be a real way to specify a breakpoint location that is between those statements.

Once the program is paused, a request can be made to look at variables, or run some more code. If a command specifies to run additional code, there may be a minimum amount of code, like a single statement, that gets run. So, debugging software might not have an easy way to pause after a partial expression is evaluated, but before the rest of the expression is evaluated. If there is a way to pause at such a point, the method might not be very standardized with other debugging software.

These limits, just described, are based on some implementations, or at least experiences that used existing implementations and performed simple steps. Other software may have different limits, or may provide some easier method of accomplishing a task. So, results may vary.

Breakpoints in GDB
Creating a breakpoint

To make a breakpoint, there are a few ways to do this. One way is to try to specify a function name, in order to pause at the start of the function. (When looking at Assembly code, there may be a few steps of assembly that get executed. Those instructions are related to handling the parameters, and that code runs before the first statement in the C function.)

b functionName

If there are multiple files, then a filename may need to be specified:

b filename.c:functionName

Another way may be:

b lineNumber

or...

b filename.c:lineNumber

A line of assembly code may be breakpointed by referring to the address:

b *0xaddressOfInstruction

e.g.:

b *0x80456789

e.g.: DJ Delorie's GDB manual: section on breakpoint commands

People using a method (a function of an object) in C++ may need use a slightly different syntax: see Ryan (Michael) Schmidt's gdb Tutorial on breakpoints: C++ methods.

More basic functionality

One a breakpoint is set, go ahead and view the breakpoints.

info breakpoints

The “Type” will be “breakpoint” (for a standard breakpoint; another possible type is a ““watchpoint”). The “Enb” column specifies whether the breakpoint is enabled.

To remove a breakpoint, find its breakpoint number, and then use:

delete breakpointNumber

To disable a breakpoint until it is manually re-enabled:

disable breakpointNumber

To disable a breakpoint for a certain number of iterations, and then to have the breakpoint be re-enabled after it was skipped for the specified number of times:

ignore breakpointNumber numberOfTimesToSkip

To re-enable a disabled breakpoint:

enable breakpointNumber

Other details

GDB also has some really neat functionality of being able to run commands automatically when a breakpoint is hit. (These are gdb commands, run from within gdb, and not necessarily commands that are part of the program that is being debugged.) Details are provided in this guide after discussions about inspecting memory, and single stepping. Be sure to not forget to check out GDB's support for breakpoint-specific automatic commands.

To create a temporary breakpoint, which is automatically deleted after the first time it is used:

Use “tbreak” instead of “break

Inspect
Conceptual overview

This allows the user to see the contents of variables, even when the code does not contain an instruction to output the value of the variable.

A simple example of an “inspect” capability is to look at the value of a specific variable.

The “inspect” command might even be able to report the results of some more advanced expressions, like the value of x*y (the value of variable x, times the value of the variable y), or even results that involve executing additional code. For example, an “inspect” command might be able to run abs(x), and then be able to report the results of running that function.

The exact capabilities and limits of an “Inspect” command may depend on just what debugging software is being used.

When using an fairly advanced piece of debugging software that is designed to support a specific programming language, the supported expressions will often use a syntax that closely resembles (and, ideally, will appear to be identical to) the syntax of the supported programming language. So, “2 **x” might mean “2 raised to the x'th power” in BASIC, or may mean “2 times *x” (and “*x” refers to a de-referenced value that is pointed to by x) in C.

Inspecting in GDB
print
Outputing a variable

To print the contents of a variable:

print variableName
Controlling format

If a format is not specified, gdb may output in a default format (possibly the last format used). For example, a number may be printed as a signed decimal (base 10) format.

The format may be controlled. For example, to show the same data in hexadecimal format:

print /x variableName

To see the details about what formats are available, help print notes that the parameter “may be preceded with /FMT, where FMT is a format letter” which is documented elsewhere. Viewing the help for the eXamine command (with “help x”) shows:

Format letters are o(octal), x(hex), d(decimal), u(unsigned decimal),
  t(binary), f(float), a(address), i(instruction), c(char) and s(string).

(The help for the eXamine command, viewable with “help x”, also describes a “repeat count” and a “size letter”. Those are not supported by the “print” command, so do not try to use one of the “size letter” options with the “print” command. The only options that are also valid for the “print” command are the options which are a “format letter”.)

Assembly registers

Assembly language users may use this technique to output a register. (The following example might be assuming AT&T syntax/flavor?)

print $eax

This may get to be slightly confusing: registers in AT&T syntax are prefaced with a percent sign. So, gdb will show the percent sign when outputting disassembled code. However, the variable names in gdb use a $. To confuse things worse, the $ has a different meaning in AT&T syntax assembly language than what it means in gdb. So, be careful with the symbols. Seeing a symbol in one part of gdb does not mean that the symbol does the same thing in another place.

Pointers

If the item being printed out is a pointer, then seeing the value of the memory address can be helpful in some cases. What is probably even more useful is seeing the contents stored in the memory that is pointed at. That can be done, but a data type must be specified. Otherwise, gdb may say “Attempt to dereference a generic pointer.” So, specify a data type. For example, if this is an integer:

print *(int *)($ebp)
Pointers with offsets

If AT&T-style disassembled code shows -0xc($ebp) (which refers to $ebp minus 0xc bytes), that would equate to:

print *(int *)($ebp - 0xc)

If AT&T-style disassembled code shows 16($ebp) (which refers to $ebp plus 0xc bytes), that would equate to:

print *(int *)($ebp + 0xc)
printf

Another approach may be to use GDB's built-in printf command:

printf "value=%d\n" , varName

The first parameter to printf needs to be a string. What printf will do is to look in the string, and figure out what else it needs to do. When it comes across a %d, it knows that there is a decimal value that it needs to print out. So, printf will expect to be given another parameter, which it will treat as a decimal value. So, the number of parameters that printf requires will depend on how many special symbols are in the string. The following shows an example of printing two registers:

printf "value=0x%x, valuetwo=%d\n" , varName, goodnum

The printf command will notice that there is a %x, so it will look for a parameter and print it out in hexadecimal. This causes varName to be printed out in hexadecimal. The next thing that printf notices is a %d, so printf will look for another parameter to be printed as a decimal. This will cause the goodnum variable to be printed out as a signed decimal.

The idea that %d is a signed decimal integer, and %x is used for hexadecimal output, may be intuitive for people who are familiar with the printf command in C. For other people, they may find that these options, and others, are documented in the help for the examine command. (Type “help x” and use a “format letter” mentioned by that text.)

x

Another popular command is examine, which is abbreviated as x.

(gdb) help x
Examine memory: x/FMT ADDRESS.
ADDRESS is an expression for the memory address to examine.
FMT is a repeat count followed by a format letter and a size letter.
Format letters are o(octal), x(hex), d(decimal), u(unsigned decimal),
  t(binary), f(float), a(address), i(instruction), c(char) and s(string).
Size letters are b(byte), h(halfword), w(word), g(giant, 8 bytes).
(More help truncated from this quoted text)
(gdb) x $ebp
0xffffd9e8:     0x0xffffda18
(gdb) x $ebp+8
0xffffd9f0:     0x0804dd50
(gdb) x 0x0804dd50
0x0804dd50 <functionName+offset>:   0x31313131
(gdb) 

This previous example showed some of the help, and then demonstrated outputing the contents of $ebp, which is a pointer to a location in the stack. More interestingly, the example outputs bytes starting at 8 bytes away from the location pointed to by $ebp. In AT&T x86 assembly syntax, that is 8($ebp). The result was a pointer to RAM; specifically the memory address 0x804dd50 (in this example). Examining that showed the actual contents in memory, which in this case contained four 0x31 characters (which happens to be the ASCII code for the digit one). In this example, it was suspected that the contents of memory was a string of seven ones. This was confirmed:

(gdb) x /xg 0x0804dd50
0x0804dd50 <functionName+offset>:   0x0031313131313131
(gdb) x /s 0x0804dd50
0x0804dd50 <functionName+offset>:   "1111111"

As you can see, this had seven 0x31 characters followed by a 0x00 character. Actually, in the first command, it looks like the 0x00 comes first. That is likely just endianness at work. Output may be compared with the following example.

(gdb) x /xh 0x0804dd56
0x0804dd56 <functionName+offset>:   0x0031

Here is an example of the “repeat count” (which is part of the “FMT”) that was referred to in the help. The variable pointing to the desired memory is $ebp (which points to the %ebp register in IA-32 x86 assembly). This shows 20 hexadecimal words that are 32 bits each.

(gdb) x /20x $ebp
0x...
(gdb) 
info registers

People who are looking at Assembly code may want to check the status of registers frequently. One way to do that is to output the values of some symbols that are named after the registers. For instance, print /x $ax

Another way is to run the following (which might show the names of the larger registers?)

info registers

(This may also be abbreviated, unambiguously, as “info reg”, or even ambiguously as “info r”. Typing “info re” and pressing tab twice will show that there is also a word “record”, but gdb will show the registers even if the command line only specifies the first letter of the last word.)

Want to know if the zero flag is set? On IA-32, this may be done by running:

info registers eflags

That will output information about just the one register. For this flags register, the value of the register is shown, as well as the abbreviations for bits that are set (to a value of 1). If the bits are cleared (to a value of zero), the abbreviation for that bit is not shown. So, if the zero flag is set, the line will say “ ZF ”.

Some registers may not be shown, including those related to floating point operations. To see additional registers:

info all-registers
display

Another neat thing that GDB supports is the “display” commands. When GDB hits a breakpoint, GDB will automatically run the display commands. This makes a convenient method of automatically showing the state of certain variables whenever the breakpoint is reached.

display variableName
info display
display
undisplay displayNumber

e.g.: The following shows removal of a display.

(gdb) display $eax
1: $eax = 123456789
(gdb) display $ecx
2: $ecx = 123456789
(gdb) display $ebx
3: $ebx = -123456789
(gdb) info display
Auto-display expressions now in effect:
Num Enb Expression
3:   y  $ebx
2:   y  $ecx
1:   y  $eax
(gdb) undisplay 2
(gdb) display $edx
4: $edx = 1
Auto-display expressions now in effect:
Num Enb Expression
4:   y  $edx
3:   y  $ebx
1:   y  $eax

The new display will show a number higher than the last created display, despite the fact that display number two stop being used. In the previous example, just shown, the second display was removed, but then the new display was still numbered display #4.

This may sound quite useful, to be able to see things automatically. However, keep in mind that the displays will show with every breakpoint. GDB's support for breakpoint-specific automatic commands may be more useful.

Some additional examples may be shown by: gdb gnu assembler

Watch
Overview

The debugging software keeps watching the program, checking if a certain thing happened. If a certain specified situation occurs, then an action (like pausing the program) is taken. Conditional breakpoints are a prime example of using a watch.

For instance, a watch might be set to have the program watch the value of a certain variable. If the variable is ever assigned a specific value, or if the variable is ever changed from having a value, then the program may be paused. Perhaps the watch simply looks to see whenever the variable changes. (So, if the variable is changed from 5 to 7, then the program is paused and a new watch is automatically created to start noticing when the program changes from a value of seven.)

The “watch” might even be able to watch for more advanced situations, like when x*y (the value of variable x, times the value of the variable y) reaches a certain value, or when abs(x) reaches a certain value.

The exact capabilities and limits of a “Watch” command may depend on just what debugging software is being used. The specific capabilities of the “watch” functionality may be quite similar to the capabilities of the “inspect” functionality.

Stepping
Overview of stepping methods

Here are some ways to step through code:

  • Single Step, to next statement
  • Step Over Function, to next statement
  • Single Step, to next Instruction
  • Step Over, to next Instruction

These are simply names that were invented by the author of this guide. Different debugging software implementations may have some slightly different names. So, for now, focus on the concept, rather than the names.

There may often be no difference between these stepping methods when the instruction being executed is sufficiently simple, such as a simple assignment statement that sets a variable to hold a value of zero.

Stepping into, or over, functions

The difference between “Single Step” or “Step Over” only matters when there is a function call. For example, after calculating some numbers, a function may specify that display_results() should be executed. (The display_results() function is a made up function, being used as an example.) “Single Step” will pause the program at the start of the display_results() function, so that the display_results() may be debugged.

On contrast, the “Step Over” will pause the program after the display_results() function call exits. So, instead of placing the temporary breakpoint at the top of the display_results() function that gets called, the temporary breakpoint is placed right after the display_results() function exits. The code will not pause during the display_results() function, unless a reason to pause (such as another breakpoint) is experienced.

The point of the “Step Over” functionality is that it allows code that is well-understood, such as a simple “display_results()” function, to be executed quickly, without needing to slowly go through that well-understood code in the debugger. After the well-understood function is completed, the main program is slowed down (paused) again, so that the programmer can focus on other code that needs more attention. This minimizes the time taken by well-understood code that doesn't need to immmediately be carefully scrutinized. Making the well-understood code run fast will reduce how much time the well-understood code is distracting the programmer from being able to focus on other code.

Statements or instructions

Consider the following code:

x = ( 3 * abs( y ) )+ (5 * pow( z , 2 ) );

(That is written in the language of C. This code takes 3 times the variable y, and adds that to five times the amount of z squared. The results are stored in a variable called x. Some of the parenthesis in that example are not required, but were added to clearly show the order that the math will occur.)

All of that is considered to be one “statement”. The statement involves multiple expressions, including multiple function calls, but it is one statement.

Debugging software may be able to recognize where statements are separated. (For example, at least as a general guideline, many modern programming languages will frequently end one statement, and start treating the next text as a new statement, when there is a semi-colon (“;”) or a right curly brace (“}”).

However, the assembly code may commonly have this be broken up into multiple instructions: two instructions to multiply, one instruction to add, and two instructions that specify that function calls should occur.

Debugging software may allow the user to advance to the next assembly instruction, rather than advancing all the way to the next statement. This can frequently be very nice when looking closely at the assembly code. Even when a programmer is not trying to work directly with the assembly code, advancing to the next assembly-level instruction can be nice. By stopping after an assembly instruction is executed, this may help a programmer to see the effects of one portion of a complex statement, before the program moves onto another portion of complex statement.

Some older text

(This may repeat the prior text; and should be reviewed/merged.)

There are generally two ways to “single step” through a (main, debugged) program. One method is to “step into” functions, and the other is to “step over” all of the details of a function.

Stepping into a function will start running a function, but will pause at the start of a function. This means that the user (of the debugging software) may then perform debugging on the function that is being called.

Stepping “over” a function will start running a function, but the “stepping” process does not cause the software to pause when the function is started. Therefore, the program will continue to run, until there is some reason to pause. (For example, if the program reaches a “break point”, the program may be paused.) However, upon exiting the called function, the program will stop. This allows the debugging software to be able to quickly run a statement, including any trusted function calls, but then to pause once that statement is completed. Once paused, the person with the debugging software may wish to inspect the value of variables, or proceed to run some more code (perhaps by stepping “into” the next function call).

Stepping in GDB

When dealing with assembly language, using the “ni” or “si” commands may be most helpful.

For higher level languages, the “n” command steps over all of the function calls in a statement, getting the program to the start of the next statement. The “s” command steps into a function.

Note that using the “n” command or the “s” command may require some details in the executable. Otherwise, gdb may print something like:

(gdb) s
Single stepping until exit from function currentFunction, which has no line number information.

(If using the “n” command instead of the “s” command, the results are the same. The same message is displayed.)

Then GDB runs the program (at full speed), possibly stopping immediately after exiting the function. (Therefore, the code will be in whatever function made the function call to this function.) Essentially, the entire remainder of the function may be treated like one statement, and so GDB simply finishes the function. (If that isn't desired, then using another breakpoint may be needed. A breakpoint can effectively stop GDB.)

(This just-described behavior may be something that gets encountered when dealing with an executable file where GDB is not finding all of the source files.?)

Some other neat things to do with debuggers
GDB: Automatic startup actions

The program will then check for an init file. A system-wide init file may be used, as well as a user-specific init-file, as well as an init file in each subdirectory. So, each time that gdb is started from a specific directory, gdb can find an init file that specifies what executable file to start loading, performing tasks establishing desired breakpoints, and performing other tasks like reporting which breakpoints exist. A user can simply get into the habit of checking that those options were successfully performed, instead of trying to remember to do each of those steps before running the program.

Further details may be seen in the official documentation about GDB Startup and Init file in current directory. Note that if there are complex/elaborate processes that are desired, such complexity may be moved to other “command files”. That concept is referred to, in more detail, in the seciton about GDB's support for breakpoint-specific automatic commands.

See also:

[#gdbkpcmd]: GDB's support for breakpoint-specific automatic commands

Sure, a person could use the “display” command to show the contents of a variable at every single breakpoint. Or, breakpoint-specific commands may be entered.

This can be great for being able to insert comments into code that isn't commented. (For example, automatically generated code from a disassembler may have comments quite sparse.)

In the following example, a program is running but is paused. A breakpoint has already been created, and the user wishes to modify an existing breakpoint which is already assigned a breakpoint number. In this example, the breakpoint being worked with is breakpoint number 4. (That implies that there were three breakpoints created earlier. This does not imply that those earlier breakpoints have not been deleted yet.)

In the following example, the user types in two printf commands that should run whenever a specific breakpoint is reached. Then, the user verifies that those commands show up related to the expected breakpoint. Finally, the user continues running a program, and the output of the printf statements may be seen.

(gdb) command 4
Type commands for breakpoint(s) 4, one per line.
End with a line saying just "end".
>printf "-= Ensures that current counter doesn't exceed max =-\n"
>printf "%eax is 0x%x, %edx is 0x%x\n" , $eax , $edx
>end
(gdb) info breakpoints
Num     Type           Disp Enb Address    What
(info about other breakpoints)
      breakpoint     keep y   0x080489ab What
        breakpoint already hit 1 time
        printf "-= Ensures that current counter doesn't exceed max =-\n"
        printf "%eax is 0x%x, %edx is 0x%x\n" , $eax , $edx
(info about other breakpoints)
(gdb) c
Breakpoint 4, 0x080489ab in fcnName ()
-= Ensures that current counter doesn't exceed max =-
%eax is 0x5678abcd, %edx is 0x0
(gdb) 

This does “clutter” the output of “info breakpoints”, but that clutter can be useful comments that describe the breakpoint. That may be quite helpful when there are a number of breakpoints. Being quickly reminded of the purpose of the breakpoint can also be helpful when the breakpoint is unexpectedly encountered. Also, key reminders about the expectations of variables may be helpful, particularly when variables get re-used (which may be common when using registers in an Assembly Language).

The parameter after the word “command” is the breakpoint number. Once GDB sees that the user wants to create a command for the breakpoint, GDB forgets about any previously defined custom commands, and then GDB asks the user for custom commands. Existing commands are not edited; they are replaced. So, if you are using an environment where “copy and paste” functionality is relatively convenient, then be sure to save a copy of any desired commands (into a conveniently accessed text file) if there is a chance that the command will need to be re-typed.

You probably don't want to make the list of commands too elaborate: keep in mind that these commands will run every single time that the breakpoint is reached. However, if there is a desire to do something complicated, it may make sense to create a GDB “command file”. To do that, type the desired commands into a text file. In GDB, use the source command to load up the text file. Note that some commands may run differently (notably, with less prompts) when they are loaded from this type of script file. Further details are mentioned in the official documentation at GDB Command files.