AMD64 Instruction-Level Debugging With dbx

Debugging at the machine-instruction level in the Oracle Developer Studio dbx command-line debugger environment becomes very handy when a software bug cannot be found easily. Usually, programs are written in high-level languages such as C, C++, or Fortran, and most of the software defects can be debugged in the dbx environment at the same high level.

However, having some knowledge of the machine-instruction level of the system on which the program is running, and using the right tool, such as dbx, can shorten the time to identify the culprit and come up with the optimal solution to fix the defect.

This article describes how to use the dbx debugger efficiently on the AMD64 architecture. It describes how to display the contents of memory at specified addresses, and how to display machine instructions. Use the regs command to print out the contents of machine registers or the print command to print out individual registers. Use the nexti, stepi, stopi, and tracei commands to debug at AMD64 machine-instruction level.

AMD64 Architecture

First let's review briefly the AMD64 architecture and see how it is different from the 32-bit x86 architecture. I describe only the materials that are relevant to this article. For an in-depth understanding of AMD64 architecture, please refer to AMD64 manuals (http://developer.amd.com) and AMD64 Application Binary Interface (ABI) (http://www.x86-64.org).

The AMD64 architecture has sixteen 64-bit general purpose registers (GPRs): RAX, RBX, RCX, RDX, RBP, RSI, RDI, RSP, R8, R9, R10, R11, R12, R13, R14, and R15. Compared to the x86 architecture, the AMD64 architecture has eight new GPRs.

The RAX, RBX, RCX, RDX, RBP, RSI, RDI, and RSP registers are used by both 32-bit and 64-bit binaries. However, in 32-bit mode, only the low 32 bits of these registers are accessible by 32-bit binaries. In the x86 architecture, these registers are EAX, EBX, ECX, EDX, EBP, ESI, EDI, and ESP, respectively.

The AMD64 architecture provides sixteen 128-bit XMM registers XMM0 through XMM15. Registers XMM0 through XMM7 are used for passing float and double parameters. The long double type is passed in memory. A long double in AMD64 architecture is 16 bytes long compared to 12 bytes in the x86 architecture. The long double type is implemented based on 80-bit extended (IEEE) standard.

The AMD64 architecture also provides eight x87 floating point registers, MMX0/FPR0 through MMX7/FPR7, each 80 bits wide.

In contrast to the 32-bit architecture in which the function parameters are passed on the stack, the 64-bit architecture has six registers available for integer parameter passing. If the number of integer parameters is more than six, the remaining parameters are passed on the stack.

The bool, char, short, int, long, long long, and pointer types are classified as integer class. For passing parameters of the integer class, the next available register of the sequence RDI, RSI, RDX, RCX, R8, and R9 is used.

Registers RBP, RBX, and R12 through R15 belong to the calling function, and the called function is required to preserve their values.

The RIP register is the instruction pointer register. In 64-bit mode, the RIP register is extended to 64 bits to support 64-bit offsets. In 32-bit x86 architecture, the instruction pointer register is the EIP register.

The return value of a function is classified based on the rules that are specified in AMD64 ABI. For instance, if the return value needs to be passed in memory, then the caller provides space for the return value and passes the address of this storage in the RDI register as if it were the first argument to the function. On return, the RAX register contains the address that has been passed by the caller in the RDI register.

Similarly, if the return type is integer, the next available register of the sequence RAX, RDX is used.

In addition to registers, each function has a frame on the run-time stack. The run-time stack grows downwards from a high address. Table 1 shows the stack organization.

Table 1 Stack Frame With Base Pointer

The RSP register is the stack pointer register and the RBP register is the frame pointer register. Stack operations make implicit use of the RSP register, and in some cases, the RBP register. The RSP register is decremented when items are pushed onto the stack, and incremented when they are popped off the stack. The RBP register points to the lowest address of the data structure that is passed from one function to another.

The 128-byte area beyond the location pointed to by the RSP register is known as red zone and is considered to be reserved. Functions can use this area for temporary data that is not needed across function calls. In particular, leaf functions can use this area for their entire stack frames, rather than adjusting the stack pointer in the prologue and the epilogue.

prologue:

pushq %rbp          / save frame pointer
movq %rsp,%rbp      / set new frame pointer
subq $48,%rsp       / allocate stack space
movq %rbx,-16(%rbp) / save callee-saved registers
movq %r12,-24(%rbp)
movq %r13,-32(%rbp)
movq %r14,-40(%rbp)
movq %r15,-48(%rbp)

There is no need to adjust the RSP stack pointer register if the red zone area is used. In other words, the subq $48,%rsp instruction is not needed in function prologue if the red zone area is used.

epilogue:

movl -4(%rbp), %eax / set up return value
movq -16(%rbp),%rbx / restore callee-saved registers
movq -24(%rbp),%r12
movq -32(%rbp),%r13
movq -40(%rbp),%r14
movq -48(%rbp),%r15
leave
ret

The C++ language has its own Application Binary Interface (ABI). The C++ ABI has well-defined rules for function parameter passing and return values. The C++ ABI rules supplement the AMD64 ABI rules; the C++ compiler has to use the C++ ABI rules for function parameter passing in addition to the AMD64 ABI rules.

dbx Commands

The following commands are documented in Oracle Solaris Studio 12.2: Debugging a Program With dbx for machine-instruction level debugging.

The Problem Statement

To demonstrate machine-instruction level debugging, let's use a real bug report that was filed against the 64-bit dbx on the AMD64 platform, including a test case. The bug report says “

On AMD64 dbx prints hex values instead of letters after strchr call:

(dbx) print strchr("hello", 'l') = 0xfffffd7fffdff742 "\xdf\xff^?\xfd\xff\xff^D"

Here is the test case:

  char *b = "hello";

  printf("%s\n", b);
  printf("%s\n", strchr("hello", 'l'));
}

There is nothing wrong with the program. The bug is in dbx.

The dbx Failure

First let's observe the normal flow of the program in the dbx environment by just stepping through the test case code.

% dbx a.out

Reading a.out
Reading ld.so.1
Reading libc.so.1
(dbx) stop in main
(2) stop in main
(dbx) run

Running: a.out
(process id 16245)
stopped in main at line 3 in file "1.c"
  3 char *b = "hello";
(dbx) next
stopped in main at line 4 in file "1.c"
  4 printf("%s\n", b);
(dbx) next
hello
stopped in main at line 5 in file "1.c"
  5 printf("%s\n", strchr("hello", 'l'));
(dbx) next
llo
stopped in main at line 6 in file "1.c"
  6 }
(dbx) next
execution completed, exit code is 4

The print statement at line 5 calls the strchr() function with two parameters. The strchr() function searches through the first parameter hello and returns a pointer to the first occurrence of the l character. Hence, the llo character string is displayed correctly by the printf statement.

Now let's reproduce the failure by calling the strchr() function directly from the dbx command line using the print command. The call command in dbx can also be used to call the strchr() function from the command line.

% dbx a.out


Reading a.out
Reading ld.so.1
Reading libc.so.1
(dbx) stop in main
(2) stop in main
(dbx) run
Running: a.out (process id 14772)
stopped in main at line 3 in file "1.c"<
  3 char *b = "hello";
(dbx) next stopped in main at line 4 in file "1.c"
  4 printf("%s\n", b);
(dbx) next hello
stopped in main at line 5 in file "1.c"
  5 printf("%s\n", strchr("hello", 'l'));
(dbx) print strchr("hello", 'l')

strchr("hello", 'l') = 0xfffffd7fffdff742 "\xdf\xff^?\xfd\xff\xff^D"

dbx prints incorrect output when the strchr() function is called by the print command. dbx should display the llo string instead of hex characters, since the call to the strchr() function is supposed to return a pointer to the first occurrence of the l character in the string hello.

The Debugging Session

Let's run the debugger with the a.out executable and stop right before the printf statement. The strchr() function is defined in the libc library and most likely is not compiled with the -g option. So there is no debugging information and we have to rely on the assembly code only.

The stopi command is used to set a breakpoint at the first machine instruction of the strchr() function.

% dbx a.out


Reading a.out
Reading ld.so.1
Reading libc.so.1
(dbx) stop in main
(2) stop in main
(dbx) run
Running: a.out (process id 15045)
stopped in main at line 3 in file "1.c"
  3 char *b = "hello";
(dbx) next
stopped in main at line 4 in file "1.c"
  4 printf("%s\n", b);
(dbx) next
hello
stopped in main at line 5 in file "1.c"
  5 printf("%s\n", strchr("hello", 'l'));
(dbx) stopi at strchr
(3) stopi at &strchr
(dbx) print strchr("hello", 'l')stopped in strchr at 0xfffffd7fff307910
0xfffffd7fff307910: strchr : movb& (%rdi),%dl

dbx stops at the first instruction of the strchr() function after the strchr() function is called from the dbx command line using the print command.

The dis command can be used to display the first portion of machine instructions for the strchr() function.

(dbx) dis strchr


0xfffffd7fff307910: strchr       : movb (%rdi),%dl 
0xfffffd7fff307912: strchr+0x0002: cmpb %dh,%dl
0xfffffd7fff307915: strchr+0x0005: je strchr+0x3f [0xfffffd7fff30794f, .+0x3a ]
0xfffffd7fff307917: strchr+0x0007: testb %dl,%dl
0xfffffd7fff307919: strchr+0x0009: je strchr+0x33 [0xfffffd7fff307943, .+0x2a ]
0xfffffd7fff30791b: strchr+0x000b: movb 0x0000000000000001(%rdi),%dl
0xfffffd7fff30791e: strchr+0x000e: mpb %dh,%dl
0xfffffd7fff307921: strchr+0x0011: je strchr+0x3c [0xfffffd7fff30794c, .+0x2b ]
0xfffffd7fff307923: strchr+0x0013: testb %dl,%dl 
0xfffffd7fff307925: strchr+0x0015: je strchr+0x33 [0xfffffd7fff307943, .+0x1e ]

The first instruction of the strchr() function is movb (%rdi),%dl, which moves the contents of the memory location pointed to by the %rdi register to the low eight bits of the %rdi register itself. The first instruction is not the pushq %rbp instruction, which means the strchr() function has no prologue. It is not a defect that the function does not have a prologue.

The debugger is stopped at the first instruction, which is the right place in the program to verify whether the input parameters are being passed correctly to the strchr() function. The strchr() function has two parameters. The first parameter is a pointer to the memory location that contains the hello character string and the second parameter is the character l. Based on the AMD64 ABI, the first and second parameters are assigned to the %rdi and %rsi registers in sequence. There are two ways to display the content of the %rdi and %rsi registers.

• You can use the print command to print the contents of the individual registers. The -flx options force dbx to display the contents of the %rdi and %rsi registers in long-hex format.

  (dbx) print -flx $rdi
  
  
  $rdi = 0xfffffd7fffdff740
  (dbx) print -flx $rsi
  $rsi = 0x6c

• You can use the regs command to display the contents of all of the AMD64 registers.

  
  
  
          (dbx) regs
  current frame: [1]
  r15         0x0000000000000000
  r14         0x0000000000000000
  r13         0x0000000000000000
  r12         0x0000000000000000
  r11         0xfffffd7fff307910
  r10         0x0000000000000000
  r9          0x0000000000010000
  r8          0xfefeff6e6b6b6467
  rdi         0xfffffd7fffdff740
  rsi         0x000000000000006c
  rbp         0xfffffd7fffdff810
  rbx         0xfffffd7fff3fb190
  rdx         0x0000000000000000
  rcx         0x000000003f570d87
  rax         0x0000000000000000
  trapno      0x0000000000000003
  err         0x0000000000000000
  rip         0xfffffd7fff307910:strchr movb (%rdi),%dl
  cs          0x000000000000004b
  eflags      0x0000000000000282
  rsp         0xfffffd7fffdff738
  ss          0x0000000000000043
  fs          0x00000000000001bb
  gs          0x0000000000000000
  es          0x0000000000000000
  ds          0x0000000000000000
  fsbase      0xfffffd7fff3a2000
  gsbase&     0xffffffff80000000

The %rdi register contains a pointer to the memory location 0xfffffd7fffdff740, which is allocated on the stack. In the normal program flow, the %rdi register contains a pointer to the memory location in the data segment. However, when dbx is asked to call a function (strchr()), dbx copies the memory location in the data segment onto the stack and passes the stack address to the %rdi register.

The contents of the memory location 0xfffffd7fffdff740 can be verified by using the examine command. The memory location should contain the hello character string.

(dbx) examine 0xfffffd7fffdff740 / 20xfffffd7fffdff740: 0x6c6c6568 0x0000006f

By looking up the ASCII table, we can verify that indeed the memory location 0xfffffd7fffdff740 contains the hello character string. The hex number 68 stands for the character h, 65 stands for the character e, 6c stands for the character l, and 6f stands for the character o.

You can use the examine command directly to display the contents of the memory location 0xfffffd7fffdff740 as a character string without referring to the ASCII table .

(dbx) examine 0xfffffd7fffdff740 / 6c0xfffffd7fffdff740: 'h' 'e' 'l' 'l' 'o' '\0'

The %rsi register contains the hex number 6c, which stands for the l character.

The other two important registers are the %rsp (the stack pointer) and %rbp (the frame pointer). The %rsp register is pointing to the top of the stack and its value is 0xfffffd7fffdff738. As you can see, this value is very close to the contents of the %rdi register, which is pointing to the memory location on the stack that contains the hello character string.

The %rbp register is the frame pointer and contains 0xfffffd7fffdff81 value. The %rbp register is not used in the strchr() function.

The contents of the run-time stack can be displayed using the examine command.

(dbx) examine 0xfffffd7fffdff738 / 32 lx

0xfffffd7fffdff738: 0xfffffd7fff220004 0x0000006f6c6c6568
0xfffffd7fffdff748: 0x0000000000000000 0x0000000000000000
0xfffffd7fffdff758: 0x0000000000000000 0xfffffd7fffdff7b0
0xfffffd7fffdff768: 0xfffffd7fff3c7e50 0x0000000000010000
0xfffffd7fffdff778: 0x0000000000000000 0x0000000000410c50
0xfffffd7fffdff788: 0x0000000000000000 0xfffffd7fffdff848
0xfffffd7fffdff798: 0x0000000000410c50 0x0000000000410c58
0xfffffd7fffdff7a8: 0xfffffd7fff3fb190 0x0000000000000000
0xfffffd7fffdff7b8: 0x0000000000000000 0xfffffd7fffdff810
0xfffffd7fffdff7c8: 0x000000000040099d 0x0000000000000000
0xfffffd7fffdff7d8: 0x0000000000000000 0x0000000000000000
0xfffffd7fffdff7e8: 0x0000000000000000 0x0000000000000000
0xfffffd7fffdff7f8: 0xfffffd7fff3fb190 0x0000000000410c50
0xfffffd7fffdff808: 0xfffffd7fffdff838 0xfffffd7fffdff820
0xfffffd7fffdff818: 0x000000000040080c 0x0000000000000000
0xfffffd7fffdff828: 0x0000000000000000 0x0000000000000001 

In fact, we can unwind the run-time stack by following the principles that we learned in the previous section (see Table 4) about the stack frame with the base pointer. For instance, the hex number 0x40080c is the address of next instruction after the callq instruction. The main function is called from the _start() function using the callq instruction.

The hex number 0x40080c is the return address that is pushed onto the stack before the call to the main() function. The instruction at address 0x40080c, push %rax, will be executed upon the completion of the main() function. In other words, the address 0x40080c will be loaded into the program counter, the %rip register, once the main function returns.

You can use the objdump utility program to dump the text section of an executable.

objdump -S a.out


00000000004007a0 <_start>:

  4007a0:    6a 00                   pushq  $0x0
  4007a2:    6a 00                   pushq  $0x0
  4007a4:    48 8b ec                mov    %rsp,%rbp
  4007a7:    48 8b fa                mov    %rdx,%rdi
  4007aa:    48 c7 c0 80 0a 41 00    mov    $0x410a80,%rax
  ...

  400806:    59                      pop    %rcx
  400807:    e8 54 01 00 00          callq  400960 <main>
  40080c:    50                      push   %rax
  40080d:    50                      push   %rax 
  ...

The first instruction of the main function is push %rbp. Hence, the previous frame pointer (0xfffffd7fffdff820) is pushed onto the stack right after the return address. Similarly, the return address (0x40099d) is pushed onto the stack when the strchr() function is called from the command line.

0000000000400960 <main>

400960:    55                      push
  400961:    48 8b                   mov    %rsp,%rbp
  400964:    48 83 ec 40             sub    $0x40,%rsp
  ...

  40099d:    b8 6c 00 00 00          mov    $0x6c,%eax
  4009a2:    0f be f0                movsbl %al,%esi
  4009a5:    48 c7 c7 68 0c 41 00    mov    $0x410c68,%rdi
  4009ac:    b8 00 00 00 00          mov    $0x0,%eax

However, the strchr() function does not have a function prologue, so the content of %rbp register stays the same when the strchr() function is called from the main() function. The content of %rbp register is the hex value 0xfffffd7fffdff810 and in turn the content of the 0xfffffd7fffdff810 address points to the previous frame pointer 0xfffffd7fffdff820.

(dbx) examine 0xfffffd7fffdff810

0xfffffd7fffdff810: 0xfffffd7fffdff820

Going forward, we single step through the machine instructions using the nexti command until we get to the instruction that returns the return value in the %rax register. We can use the dis command to display the last portion of machine instructions for the strchr() function.

(dbx) dis


0xfffffd7fff307941: strchr+0x0031: jne strchr [0xfffffd7fff307910, .-0x31 ]
0xfffffd7fff307943: strchr+0x0033: xorl %eax,%eax
0xfffffd7fff307945: strchr+0x0035: ret
0xfffffd7fff307946: strchr+0x0036: incq %rdi
0xfffffd7fff307949: strchr+0x0039: incq %rdi
0xfffffd7fff30794c: strchr+0x003c: incq %rdi
0xfffffd7fff30794f: strchr+0x003f: movq %rdi,%rax
0xfffffd7fff307952: strchr+0x0042: ret
0xfffffd7fff307953: strchr+0x0043: addb %al,(%rax)
(dbx) nexti
stopped in strchr at 0xfffffd7fff307949
0xfffffd7fff307949:
strchr+0x0039:
incq %rdi
(dbx) nexti
stopped in strchr at 0xfffffd7fff30794c
0xfffffd7fff30794c: strchr+0x003c: incq %rdi
(dbx) nexti
stopped in strchr at 0xfffffd7fff30794f
0xfffffd7fff30794f: strchr+0x003f: movq %rdi,%rax
(dbx) nexti
stopped in strchr at 0xfffffd7fff307952
0xfffffd7fff307952: strchr+0x0042: ret

Based on the description of the strchr() function, at the end it is supposed to return a pointer to the first occurrence of the l character in the string hello. We can verify the correctness of the strchr() function by examining the contents of the %rax register.

(dbx) examine $rax / 4c0xfffffd7fffdff742: 'l' 'l' 'o' '\0'

Indeed, the value of the %rax register is a pointer to the memory location 0xfffffd7fffdff742, which is allocated on the stack and contains the llo character string.

We have verified that the strchr() function works correctly and returns a pointer to the llo character string in the %rax register. So the problem must be with what dbx does internally after it finishes calling the strchr() function. Fast forward, after calling a user function, dbx always calls the fflush() function to flush the output stream. The fflush() function takes one parameter, which is a pointer to the FILE data structure.

fflush - flush a stream

#include <stdio.h>
int fflush(FILE *stream);

You can use the dis command to display the machine instructions for the fflush function.

(dbx) dis fflush


0xfffffd7fff33dca0: fflush:        pushq %rbp
0xfffffd7fff33dca1: fflush+0x0001: movq %rsp,%rbp
0xfffffd7fff33dca4: fflush+0x0004: movq %rbx,0xfffffffffffffff0(%rbp)
0xfffffd7fff33dca8: fflush+0x0008: movq %r12,0xfffffffffffffff8(%rbp)
0xfffffd7fff33dcac: fflush+0x000c: subq $0x0000000000000010,%rsp
0xfffffd7fff33dcb0: fflush+0x0010: testq %rdi,%rdi
0xfffffd7fff33dcb3: fflush+0x0013: movq %rdi,%rbx

Let's go over the fflush function prologue:

pushq %rbp

Store the previous frame pointer on the stack.

movq %rsp, %rbp"

Store the value of the %rsp register or the previous stack pointer into the %rbp register. This value is the new frame pointer for the fflush() function.

movq %rbx,0xfffffffffffffff0(%rbp)

movq %r12,0xfffffffffffffff8(%rbp)

The %rbx register and the %r12 register are callee-saved registers. The fflush() function must preserve the contents of these registers on the stack for the caller function so they can be restored later in the function epilogue just before exiting the function.

sub $0x0000000000000010,%rsp

Adjust the stack pointer for the fflush() function.

The stopi command is used to stop at the first instruction of fflush() function.

(dbx) stopi at fflush

(dbx) cont
dbx: Call to 'strchr' completed. Going back to previous command
interpreter
stopped in fflush at 0xfffffd7fff33dca0

0xfffffd7fff33dca0: fflush: pushq    %rbp

dbx stops at the first instruction of the fflush

Let's display the %rdi, %rbp, and %rspregisters. The %rdi register contains a pointer to the FILE data structure.

(dbx) print -flx $rdi


$rdi = 0xfffffd7fff37f0a0
(dbx) print -flx $rbp
$rbp = 0xfffffd7fffdff810
(dbx) print -flx $rsp
$rsp = 0xfffffd7fffdff748

We step through the function prologue and print the %rsp and %rbp registers again.

(dbx) stepi


stopped in fflush at 0xfffffd7fff33dca1
0xfffffd7fff33dca1: fflush+0x0001: movq %rsp,%rbp
(dbx) stepi
stopped in fflush at 0xfffffd7fff33dca4
0xfffffd7fff33dca4: fflush+0x0004: movq %rbx,0xfffffffffffffff0(%rbp)
(dbx) stepi
stopped in fflush at 0xfffffd7fff33dca8
0xfffffd7fff33dca8: fflush+0x0008: movq %r12,0xfffffffffffffff8(%rbp)
(dbx) stepi
stopped in fflush at 0xfffffd7fff33dcac
0xfffffd7fff33dcac: fflush+0x000c: subq $0x0000000000000010,%rsp
(dbx) stepi
stopped in fflush at 0xfffffd7fff33dcb0
0xfffffd7fff33dcb0: fflush+0x0010: testq %rdi,%rdi
(dbx) print -flx $rbp
$rbp = 0xfffffd7fffdff740
(dbx) print -flx $rsp

$rsp = 0xfffffd7fffdff730

If you can recall from previous section, the run-time stack grows downwards from high address. By careful examination of the %rsp register and comparing its value (0xfffffd7fffdff730) with the last value of the %rsp register (0xfffffd7fffdff738) in the strchr() function, it becomes obvious that the space that is allocated on the stack for the fflush() function overlaps with the space for the strchr() function.

The 0xfffffd7fffdff738 value is right between the value of the %rbp register (0xfffffd7fffdff740) and the value of the %rsp register (0xfffffd7fffdff730) of the fflush() function. Therefore, the fflush() function overwrites the contents of the run-time stack for the strchr() function, which explains why the print strchr("hello", 'l') command displays garbage instead of the llo character string.

The fix for the dbx debugger is to preserve the contents of the run-time stack just before the call to the fflush() function and restore it just before returning to the print command.

In Conclusion

In general, low-level debugging requires the user to have some kind of knowledge about the system on which the program is executing. But once necessary knowledge is learned, even the most difficult bugs can be detected using the low-level debugging techniques and using the right tool, such as dbx