Debugging In Linux

From JaxHax
Revision as of 18:05, 20 May 2015 by Travis (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Overview

This class was taught at Jax LUG on 05/20/2015. This class gives a simple overview of some common debugging tools in linux and some usage and use cases.


What is debug?

Debugging is the process of tracking down and removing undesired behavior from a program that is occurring as a result of faulty logic. An example of this can be something such as:

  • A use-after-free bug in which you free and object and try to access it again after it is freed which can cause unpredicable behavior.
  • Declaring a variable and using it before initalizing it which can result in an unknown value already being in the memory space of the variable.
  • Buffer Overflows - Moving data that is larger than the buffer it is going to.
  • "Off-By-One" or "Fence Post" errors.
  • Weird "Optimization" errors.


Why debug?

Debugging can be tendious at times, so why bother with it if the source code is available. Well, sometimes the source code isn't available. Some people like to package binaries and keep it closed sourced. Other times you will have the source code and it has a problem that isn't jumping out at you during code review but you experince the problem the program runs and it might be an easier route to determine why it happens dynamically at runtime rather than review time.


Debugging Tools

This is not a complete list, but here is a list of some simple linux debug tools we will cover:

  • strace
  • ltrace
  • objdump
  • gdb
  • peda

strace

strace is a great high level debugging tool that traces out the programs syscalls. It will show syscalls as they are made with the options passed.

Let's take this example x86 Code:

;=========================================
; Program: hello1.asm
;
; Purpose: to print hello world and exit!
;=========================================
global _start
 
section .text
_start:
    mov eax, 4       ; write() syscall. 
    mov ebx, 1       ; File descript of where to write(1 = STDOUT)
    mov ecx, message ; Pointer to our string to print
    mov edx, 13      ; Length of string to print
    int 0x80         ; Invoke kernel
 
    mov eax, 1       ; exit() syscall
    mov ebx, 0       ; Return code when exiting (return 0)
    int 0x80         ; Invoke kernel
 
section .bbs
    message: db "Hello World!",0x0a   ; Our string we wish to print.


So the code above will invoke the write() system call and print hello world to the screen, then exit using the exit() system call with a return code of zero. Using strace it is REALLY EASY to see what this program does which makes this a great program to demo it with. First let's just run it normally.

[user@host] $ ./hello1
Hello World!


That's a normal run. Now let's invoke it with strace and see what happens.

[user@host] $ strace ./hello1
execve("./hello1", ["./hello1"], [/* 39 vars */]) = 0
[ Process PID=1649 runs in 32 bit mode. ]
write(1, "Hello World!\n", 13Hello World!
)          = 13
_exit(0)                                = ?
+++ exited with 0 +++
[user@host] $


So let's review what we got. The first two lines can be ignored. They just give us a little information about starting the program and attaching to it. next is the line:

write(1, "Hello World!\n", 13Hello World!
)          = 13


This line is telling us the system call 'write(1, "Hello World!\n", 13)' was made [It printed the actual string before it terminated the call string] and at the end shows "= 13" which is the return value. Write() will return the number of bytes it wrote. The next string is the system call 'exit(0)' which is simple enough to follow. It should be noted as well that the output from strace is going to stderr, which makes it easy to log to a file or view exclusively. So for a cleaner example we can run the program piping stdout to /dev/null and only see the stderr output:

[user@host] $ strace ./hello1 > /dev/null
execve("./hello1", ["./hello1"], [/* 39 vars */]) = 0
[ Process PID=2751 runs in 32 bit mode. ]
write(1, "Hello World!\n", 13)          = 13
_exit(0)                                = ?
+++ exited with 0 +++
[user@host] $


Another example well will pipe the strace output to a log file instead:

[user@host] $ strace ./hello1 2> /tmp/test
Hello World!
 
[user@host] $ cat /tmp/test
execve("./hello1", ["./hello1"], [/* 39 vars */]) = 0
[ Process PID=2786 runs in 32 bit mode. ]
write(1, "Hello World!\n", 13)          = 13
_exit(0)                                = ?
+++ exited with 0 +++
 
[user@host] $

As you can see this works great with x86 assembly. But with C it can get kinda busy due to the way C does memory management. Take this simple C Hello World program

#include <stdio.h>
 
int main() {
	printf("Hello World!\n");
	return 0;
}


If we run it, it behaves almost exactly like the x86 program:

[user@host] $ ./helloworld 
Hello World!


However, when we run it with strace we will see a HUGE Difference:

[user@host] $ strace ./helloworld 
execve("./helloworld", ["./helloworld"], [/* 39 vars */]) = 0
[ Process PID=2382 runs in 32 bit mode. ]
brk(0)                                  = 0x8100000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffffffff77c6000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=165726, ...}) = 0
mmap2(NULL, 165726, PROT_READ, MAP_PRIVATE, 3, 0) = 0xfffffffff779d000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/i686/cmov/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\3\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\300\233\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1738492, ...}) = 0
mmap2(NULL, 1743484, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xfffffffff75f3000
mmap2(0xf7797000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a4000) = 0xfffffffff7797000
mmap2(0xf779a000, 10876, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xfffffffff779a000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffffffff75f2000
set_thread_area({entry_number:-1, base_addr:0xf75f2940, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0 (entry_number:12)
mprotect(0xf7797000, 8192, PROT_READ)   = 0
mprotect(0xf77ea000, 4096, PROT_READ)   = 0
munmap(0xf779d000, 165726)              = 0
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xfffffffff77c5000
write(1, "Hello World!\n", 13Hello World!
)          = 13
exit_group(0)                           = ?
+++ exited with 0 +++


As you can see, A lot of extra stuff happens, loading of libraries, lots of memory management calls, pretty busy compared to it's x86 brother. In a large program this could make the data almost unusable. However this brings us to our next tool which was made to handle this problem.


ltrace

strace with C programs can get busy with memory management calls. However there is a better trace tool for that called ltrace. ltrace traces library calls rather than system calls. In C the results of this is much cleaner:

[user@host] $ ltrace ./helloworld 
__libc_start_main(0x80483fb, 1, 0xffb00674, 0x8048430 <unfinished ...>
puts("Hello World!"Hello World!
)                         = 13
+++ exited (status 0) +++
 
[user@host] $


As you can see this shows us what happened in a much cleaner fashion. we see our puts() call and the exit status. Much closer to what happened in the c code. Let's actually look at this is a real world application. For this example we will look at using ltrace on a wargame challenge from OverTheWire.org. This is the behemoth0 program challenge which we have a program with no C code. Let's run it and see how it behaves:

behemoth0@melinda:/behemoth$ ./behemoth0 
Password: test
Access denied..
behemoth0@melinda:/behemoth$


Hmmmm... Okay, so it prompts us for a password, so we gave it "test" and it returns "Access denied..". Let's try this again but this time using ltrace :-)

behemoth0@melinda:/behemoth$ ./behemoth0 
Password: test
Access denied..
behemoth0@melinda:/behemoth$ ltrace ./behemoth0 
__libc_start_main(0x80485a2, 1, 0xffffd734, 0x8048690 <unfinished ...>
printf("Password: ")                         = 10
__isoc99_scanf(0x804876c, 0xffffd64b, 0xffffd640, 0x80482d2Password: test
) = 1
strlen("OK^GSYBEX^Y")                        = 11
strcmp("test", "eatmyshorts")                = 1
puts("Access denied.."Access denied..
)                      = 16
+++ exited (status 0) +++
behemoth0@melinda:/behemoth$


There is one line here that really jumps out at us:

strcmp("test", "eatmyshorts")                = 1


It compares my password against "eatmyshorts". Let's try running the program again with that password instead:

behemoth0@melinda:/behemoth$ ./behemoth0 
Password: eatmyshorts
Access granted..
$ whoami
behemoth1
$ exit
behemoth0@melinda:/behemoth$


Nice!!! Seems it was much happier with that. That's one example. Another would be to detect a use after free bug. You can log out to a file the syscalls and libcalls till the error occurs and compare notes.


objdump

objdump is a good tool to see the disassembly of a binary. If it is a program you can run it with the -d to get a dump of the executable parts of the program or -D to get a dump of all sections of the program. Let's look at this against the compiled x86 hello world again just dumping executable sections using -d:

[user@host] $ objdump -d hello1
 
hello1:     file format elf32-i386
 
 
Disassembly of section .text:
 
08048060 <_start>:
 8048060:	b8 04 00 00 00       	mov    $0x4,%eax
 8048065:	bb 01 00 00 00       	mov    $0x1,%ebx
 804806a:	b9 82 80 04 08       	mov    $0x8048082,%ecx
 804806f:	ba 0d 00 00 00       	mov    $0xd,%edx
 8048074:	cd 80                	int    $0x80
 8048076:	b8 01 00 00 00       	mov    $0x1,%eax
 804807b:	bb 00 00 00 00       	mov    $0x0,%ebx
 8048080:	cd 80                	int    $0x80


Now let's try to dump all the sections with -D:

[user@host] $ objdump -D hello1
 
hello1:     file format elf32-i386
 
 
Disassembly of section .text:
 
08048060 <_start>:
 8048060:	b8 04 00 00 00       	mov    $0x4,%eax
 8048065:	bb 01 00 00 00       	mov    $0x1,%ebx
 804806a:	b9 82 80 04 08       	mov    $0x8048082,%ecx
 804806f:	ba 0d 00 00 00       	mov    $0xd,%edx
 8048074:	cd 80                	int    $0x80
 8048076:	b8 01 00 00 00       	mov    $0x1,%eax
 804807b:	bb 00 00 00 00       	mov    $0x0,%ebx
 8048080:	cd 80                	int    $0x80
 
Disassembly of section .bbs:
 
08048082 <message>:
 8048082:	48                   	dec    %eax
 8048083:	65 6c                	gs insb (%dx),%es:(%edi)
 8048085:	6c                   	insb   (%dx),%es:(%edi)
 8048086:	6f                   	outsl  %ds:(%esi),(%dx)
 8048087:	20 57 6f             	and    %dl,0x6f(%edi)
 804808a:	72 6c                	jb     80480f8 <message+0x76>
 804808c:	64 21 0a             	and    %ecx,%fs:(%edx)
[user@host] $


You may have noticed the section contains <message> which it's instructions look like gibberish but it's because it's trying to read the ASCII values of the string "Hello world!\n" as code. Now what if we had some code that was acquired from outside a program? Say we had a packet capture and saw an exploit attempt and we extracted the shell code. How can we use objdump to dump it for us? Let's pretend we got the following shellcode:

\x6a\x02\x58\xcd\x80\xeb\xf9


How can we determine what these 7 bytes would have done if they ran on our system? Well we can use objdump here. First we need to dump the bytes to a file. We'll use echo with the -ne switches (-e tells it to interpret the escape characters, -n tells it to omit the new line character at the end) to dump it out to a tmp file.

[user@host] $ echo -ne "\x6a\x02\x58\xcd\x80\xeb\xf9" > /tmp/shellcode.bin


Then use objdump to disassemble it. objdump will need some special switch to understand how to read it. We will use "-D" since it doesn't have sections, we want it to read everything. we will use "-b binary" to tell it this is just loose binary. and finally "-m i386" to tell it the architecture since there is no ELF header data to define that for it. Let's dump it:

[user@host] $ objdump -D -b binary -m i386 /tmp/shellcode.bin 
 
/tmp/shellcode.bin:     file format binary
 
 
Disassembly of section .data:
 
00000000 <.data>:
   0:	6a 02                	push   $0x2
   2:	58                   	pop    %eax
   3:	cd 80                	int    $0x80
   5:	eb f9                	jmp    0x0
 
[user@host] $


From here we can see it puts the value 2 into eax (push 0x2, pop eax) and invokes the kernel (int 0x80); this will call the fork() syscall. Then we jump back to line zero (jmp 0x0) and re-run the fork() syscall. This is a 7 byte forkbomb! A sort of DoS (Denial of Service) payload. If this ran successfully it would have likely made the system go belly up unless the system limits PIDs to processes.


gdb

GDB is the Gnu DeBugger. It's meant to be the de facto debugger by the GNU Project. This tool is a powerful debugger with a lot of features you would expect from a debugger. It will disassemble code for you, attempt to dereference jumps and calls to where they are going, it allows you to set breakpoints, review memory and format the way it's shown to us, view registers and flags. Here I will attempt to cover how to use a few of these features. To demo gdb we will use the following code, it isn't buggy but does enough we can demo some features of the debugger with it:

///////////////////////////////////////////////////////////////////
//
// Program: greetz.c
//
// Author: Travis Phillips
//
// Date: 05/20/2015
//
// Purpose: Simple program that will take a name as an
//          argument and greet them. This program was
//          designed to be a simple example for a presentation
//          at JaxLUG on 05/22/2015 on using GDB debugger to
//          examine a program. Presentation notes can be found
//          on http://wiki.jaxhax.org/index.php/Debugging_In_Linux
//
///////////////////////////////////////////////////////////////////
#include <stdio.h>
#include <stdlib.h>
 
//////////////////////////////////////////
// printUsage Function that will print
// the program usage and exit with a
// return code of 1.
//////////////////////////////////////////
void printUsage(char *progName){
	////////////////////////////////////////////////
	// Escape sequences that will color our output
	////////////////////////////////////////////////
	char *Yellow = "\033[33;1m";
	char *Red = "\033[31;1m";
	char *Normal = "\033[0m";
 
	////////////////////////////////////////////////
	// a printf call with a lot of parameters ;-)
	////////////////////////////////////////////////
	printf("\n\t%s[*] Usage%s: %s %s<Name>%s\n\n", Yellow, Normal, progName, Red, Normal);
	exit(1);
}
 
int main(int argc, char **argv) {
 
	/////////////////////////////////////////////
	// Check if the user provided an argument.
	// if not we will call printUsage.
	/////////////////////////////////////////////
	if ( argc != 2 ) {
		printUsage(argv[0]);
	}
 
	/////////////////////////////////////////////
	// If they provided an extra argument then
	// we will greet that name given by the user
	/////////////////////////////////////////////
	printf("\n\tGreetings %s!\n\n", argv[1]);
 
	/////////////////////////////////////////////
	// Pause for drama effect...
	/////////////////////////////////////////////
	printf("Press enter to continue...\n");
	getchar();
 
	return 0;
}


And a quick example of compiling it and running it:

[user@host] $ gcc -m32 greetz.c -o greetz
[user@host] $ ./greetz
 
	[*] Usage: ./greetz <Name>
 
[user@host] $ ./greetz Travis
 
	Greetings Travis!
 
Press enter to continue...


gdb: Loading a program

To load a program in gdb you would simply invoke "gdb" with the program name as an argument. If that the program being debugged requires any arguments, omit them as you will provide them when you run the application in GDB.

[user@host] $ gdb ./greetz
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./greetz...(no debugging symbols found)...done.
(gdb)


It was kinda verbose on startup so we can quiet it down a bit using the -q switch if we want too:

[user@host] $ gdb -q ./greetz
Reading symbols from ./greetz...(no debugging symbols found)...done.
(gdb)


Either way will throw us into a (gdb) prompt. This means it is waiting for our commands. At this point the program is only loaded in memory but is not running yet and won't run till we tell it too. You can use this time to disassemble and analyis the program statically and also set an breakpoints or settings you'll need.


gdb: Running a program

One of the most basic things we may want to do is run the program. For this there is a gdb command aptly named "run". Now let's just run the program in gdb:

(gdb) run
Starting program: /home/owner/programming/c/greetz 
 
	[*] Usage: /home/owner/programming/c/greetz <Name>
 
[Inferior 1 (process 23774) exited with code 01]
(gdb)


Simple enough. We got the syntax usage message. This happened because we didn't provide an argument. arguments can be provided by just tacking them on after the run command, much like you would do when you're just running the program from the commandline.

(gdb) run Travis
Starting program: /home/owner/programming/c/greetz Travis
 
	Greetings Travis!
 
Press enter to continue...
 
[Inferior 1 (process 23782) exited normally]
(gdb)


As you can see when it runs the program it shows you the the actual command line it's running. In the case above "Starting program: /home/owner/programming/c/greetz Travis". It also shows the exit status when it terminates. In the first example, it exited with the status code of 1 which can be seen on the line "[Inferior 1 (process 23774) exited with code 01]" where the second exited with status code 0 which is a normal exit code. This is shown by the "[Inferior 1 (process 23782) exited normally]".


gdb: Disassembling the program

In gdb, you my want to look at the program instructions. GDB can disassemble instructions for you via the "disassemble" command. This command if ran by itself would disassemble the current function your in when the program is running (presumably when your program has stopped at a breakpoint). You can also provide a function name as well and it will disassemble that function for you. In C the main function is what will run first. So let's check that out:

(gdb) disassemble main
Dump of assembler code for function main:
   0x080484cf <+0>:	lea    0x4(%esp),%ecx
   0x080484d3 <+4>:	and    $0xfffffff0,%esp
   0x080484d6 <+7>:	pushl  -0x4(%ecx)
   0x080484d9 <+10>:	push   %ebp
   0x080484da <+11>:	mov    %esp,%ebp
   0x080484dc <+13>:	push   %ebx
   0x080484dd <+14>:	push   %ecx
   0x080484de <+15>:	mov    %ecx,%ebx
   0x080484e0 <+17>:	cmpl   $0x2,(%ebx)
   0x080484e3 <+20>:	je     0x80484f6 <main+39>
   0x080484e5 <+22>:	mov    0x4(%ebx),%eax
   0x080484e8 <+25>:	mov    (%eax),%eax
   0x080484ea <+27>:	sub    $0xc,%esp
   0x080484ed <+30>:	push   %eax
   0x080484ee <+31>:	call   0x804848b <printUsage>
   0x080484f3 <+36>:	add    $0x10,%esp
   0x080484f6 <+39>:	mov    0x4(%ebx),%eax
   0x080484f9 <+42>:	add    $0x4,%eax
   0x080484fc <+45>:	mov    (%eax),%eax
   0x080484fe <+47>:	sub    $0x8,%esp
   0x08048501 <+50>:	push   %eax
   0x08048502 <+51>:	push   $0x8048609
   0x08048507 <+56>:	call   0x8048330 <printf@plt>
   0x0804850c <+61>:	add    $0x10,%esp
   0x0804850f <+64>:	sub    $0xc,%esp
   0x08048512 <+67>:	push   $0x804861b
   0x08048517 <+72>:	call   0x8048350 <puts@plt>
   0x0804851c <+77>:	add    $0x10,%esp
   0x0804851f <+80>:	call   0x8048340 <getchar@plt>
   0x08048524 <+85>:	mov    $0x0,%eax
   0x08048529 <+90>:	lea    -0x8(%ebp),%esp
   0x0804852c <+93>:	pop    %ecx
   0x0804852d <+94>:	pop    %ebx
   0x0804852e <+95>:	pop    %ebp
   0x0804852f <+96>:	lea    -0x4(%ecx),%esp
   0x08048532 <+99>:	ret    
End of assembler dump.
(gdb)


Here you can see the code that actually runs the main() function of our program. The formatting of the lines is basically as follows:

   memory_address <+offset_from_function_start>:    x86_instruction


If you understand x86 assembly it is really helpful, but even if not you can see the CALLs dereferenced to their functions (e.g. call 0x804848b <printUsage>, call 0x8048330 <printf@plt>, call 0x8048350 <puts@plt>). We also had an if statement that determined if we would invoke printUsage(), these are usually translated to compare and jump instructions. In this case we had these two lines:

   0x080484e0 <+17>:	cmpl   $0x2,(%ebx)
   0x080484e3 <+20>:	je     0x80484f6 <main+39>

The first command will run a compare to check if the argument count is equal to 2. If it is then we will jump down to main+39 which will skip the call to printUsage().


If you notice some of the calls end with "@plt" that is because those functions are provided by the C libraries and are not a part of our program. Our compiler when it builds the program will make these small functions in the plt (Procedure Linking Table) that will look up the correct address for these functions at run time when the library is in memory. printUsage doesn't end with "@plt" because it is a part of our program, we wrote it into our code. Now let's disassemble that function just to give another example of the disassemble command

(gdb) disassemble printUsage 
Dump of assembler code for function printUsage:
   0x0804848b <+0>:	push   %ebp
   0x0804848c <+1>:	mov    %esp,%ebp
   0x0804848e <+3>:	sub    $0x18,%esp
   0x08048491 <+6>:	movl   $0x80485d0,-0xc(%ebp)
   0x08048498 <+13>:	movl   $0x80485d8,-0x10(%ebp)
   0x0804849f <+20>:	movl   $0x80485e0,-0x14(%ebp)
   0x080484a6 <+27>:	sub    $0x8,%esp
   0x080484a9 <+30>:	pushl  -0x14(%ebp)
   0x080484ac <+33>:	pushl  -0x10(%ebp)
   0x080484af <+36>:	pushl  0x8(%ebp)
   0x080484b2 <+39>:	pushl  -0x14(%ebp)
   0x080484b5 <+42>:	pushl  -0xc(%ebp)
   0x080484b8 <+45>:	push   $0x80485e8
   0x080484bd <+50>:	call   0x8048330 <printf@plt>
   0x080484c2 <+55>:	add    $0x20,%esp
   0x080484c5 <+58>:	sub    $0xc,%esp
   0x080484c8 <+61>:	push   $0x1
   0x080484ca <+63>:	call   0x8048370 <exit@plt>
End of assembler dump.
(gdb)

This command can provide a lot of insight to the program itself and you can review this even before you run the program. But the real power of the debugger goes into being able to examine the program right before it executes an instruction.


gdb: Setting breakpoints

Breakpoints provide a way to pause execution of a program at a certain instruction of the program. When the breakpoint is hit, it pauses execution BEFORE execution of the instruction it stopped on. To set a breakpoint you can use the "break" command. It takes an argument which can be a:

  • function name [example: break main]
  • a memory offset prepended with a "*" [example: break *0x080484bd]
  • +[num] or -[num] to set a break point at an offset from current breakpoint [example: break +50]

so let's say we want to set a breakpoint on the start of main:

(gdb) break main
Breakpoint 1 at 0x80484de
(gdb)


It will return a breakpoint number, in this case breakpoint 1. if we run the program now, it will break on our breakpoint and tell us which number breakpoint paused it:

(gdb) run
Starting program: /home/owner/programming/c/greetz 
 
Breakpoint 1, 0x080484de in main ()
(gdb)

Personally, I like to use the memory instructions. The "*" you put before it is like you would put before a pointer in C. Its preference but I think its easy to copy and paste the addresses in the break command.


gdb: Viewing the breakpoints

So let's say you set a few breakpoints:

(gdb) break *0x080484e3
Breakpoint 1 at 0x80484e3
(gdb) break *0x080484ee
Breakpoint 2 at 0x80484ee
(gdb) break *0x08048507
Breakpoint 3 at 0x8048507
(gdb) break *0x08048517
Breakpoint 4 at 0x8048517
(gdb)


and now you want to know how you can see what number relates to what. How can you check that? There is a command you can use called "info". Info does a lot of things but if you want to see your breakpoints you can use "info breakpoints" to check them.

(gdb) info breakpoints 
Num     Type           Disp Enb Address    What
1       breakpoint     keep y   0x080484e3 <main+20>
2       breakpoint     keep y   0x080484ee <main+31>
3       breakpoint     keep y   0x08048507 <main+56>
4       breakpoint     keep y   0x08048517 <main+72>
(gdb)


gdb: Deleting breakpoints

Say we decide we no longer want to have breakpoint number 1 around anymore. we can delete it using the "del" command, del takes an argument of the break number to delete one breakpoint.

(gdb) info breakpoints 
Num     Type           Disp Enb Address    What
1       breakpoint     keep y   0x080484e3 <main+20>
2       breakpoint     keep y   0x080484ee <main+31>
3       breakpoint     keep y   0x08048507 <main+56>
4       breakpoint     keep y   0x08048517 <main+72>
(gdb) del 1
(gdb) info breakpoints 
Num     Type           Disp Enb Address    What
2       breakpoint     keep y   0x080484ee <main+31>
3       breakpoint     keep y   0x08048507 <main+56>
4       breakpoint     keep y   0x08048517 <main+72>
(gdb)


If no argument is provided, it will prompt you if you want to delete all breakpoints.

(gdb) info breakpoints 
Num     Type           Disp Enb Address    What
2       breakpoint     keep y   0x080484ee <main+31>
3       breakpoint     keep y   0x08048507 <main+56>
4       breakpoint     keep y   0x08048517 <main+72>
(gdb) del
Delete all breakpoints? (y or n) y
(gdb) info breakpoints 
No breakpoints or watchpoints.
(gdb)


gdb: Continue

Once you hit a breakpoint there are a few things you can do. This section covers continuing. Using the command "continue" will make gdb resume execution of the program.


gdb: Step Instruction

Another thing that you can do when you hit a breakpoint is stepping a single instruction. Using the command "stepi" or "si" for short, you can step one instruction at which point it will pause again like it just hit another breakpoint. It is important to note that this instruction will follow "CALL" instructions.


gdb: Next Instruction

Another option is the next instruction command "nexti" or "ni". This command does about the same thing as "si" except it won't follow a "CALL" instruction.


gdb: Examine Memory

There are times where you will want to examine what is going on in memory of a program. This can be useful as a program will often push pointers to the stack to data before making a CALL instruction. Examining memory in gdb is done with the command "x/[format] [pointer]". Let's take a look at this section of code:

   0x08048512 <+67>:	push   $0x804861b
   0x08048517 <+72>:	call   0x8048350 <puts@plt>


As you can see on line <+67> it pushes some pointer to the stack for the puts() call. This should be a string value so to examine a string value we will use "x/s" and give it the address pointer.

(gdb) x/s 0x804861b
0x804861b:	"Press enter to continue..."
(gdb)


As mentioned earlier with the "x/[format]", there are many types of formats that can be used to examine memory. the formats can go in a format of "nfu" which is [n]umber, [f]ormat, and [u]nit respectively. The n part can be a number to indictate how many of whatever format you choose you want to see.


Formats
Expression What it does
s A null-terminated string
x A hexidecimal representation
i A Machine instruction representation


Units
Expression What it does
b Bytes
h Halfwords (two bytes).
w Words (four bytes).
g Giant Words (eight bytes).


So with this we can see there are a lot of ways we can view the data of that memory address. We choose to view it as a string, but we could have viewed it as hex bytes as well. Let's view it with n being 27 (string is 26 long + 1 for null byte).


(gdb) x/s 0x804861b
0x804861b:	"Press enter to continue..."
(gdb) x/27xb 0x804861b
0x804861b:	0x50	0x72	0x65	0x73	0x73	0x20	0x65	0x6e
0x8048623:	0x74	0x65	0x72	0x20	0x74	0x6f	0x20	0x63
0x804862b:	0x6f	0x6e	0x74	0x69	0x6e	0x75	0x65	0x2e
0x8048633:	0x2e	0x2e	0x00
(gdb)


gdb: Examining the stack

To examine the stack we will still use the "x/[format]" command, but we will use a value that reads from ESP (which is the stack pointer, it points to the top of the stack). The value to access this address is "$sp". So say we wanted to examine the stack before calling printf() in the printUsage() function, we could do the following:

(gdb) disassemble printUsage 
Dump of assembler code for function printUsage:
   0x0804848b <+0>:	push   %ebp
   0x0804848c <+1>:	mov    %esp,%ebp
   0x0804848e <+3>:	sub    $0x18,%esp
   0x08048491 <+6>:	movl   $0x80485d0,-0xc(%ebp)
   0x08048498 <+13>:	movl   $0x80485d8,-0x10(%ebp)
   0x0804849f <+20>:	movl   $0x80485e0,-0x14(%ebp)
   0x080484a6 <+27>:	sub    $0x8,%esp
   0x080484a9 <+30>:	pushl  -0x14(%ebp)
   0x080484ac <+33>:	pushl  -0x10(%ebp)
   0x080484af <+36>:	pushl  0x8(%ebp)
   0x080484b2 <+39>:	pushl  -0x14(%ebp)
   0x080484b5 <+42>:	pushl  -0xc(%ebp)
   0x080484b8 <+45>:	push   $0x80485e8
   0x080484bd <+50>:	call   0x8048330 <printf@plt>
   0x080484c2 <+55>:	add    $0x20,%esp
   0x080484c5 <+58>:	sub    $0xc,%esp
   0x080484c8 <+61>:	push   $0x1
   0x080484ca <+63>:	call   0x8048370 <exit@plt>
End of assembler dump.
(gdb) break *0x080484bd
Breakpoint 1 at 0x80484bd
(gdb) run
Starting program: /home/owner/programming/c/greetz 
 
Breakpoint 1, 0x080484bd in printUsage ()
(gdb) x/s 0x080485e8
0x80485e8:	"\n\t%s[*] Usage%s: %s %s<Name>%s\n\n"
(gdb) x/s 0x080485d0
0x80485d0:	"\033[33;1m"
(gdb) x/s 0x080485e0
0x80485e0:	"\033[0m"
(gdb) x/s 0xffffd572
0xffffd572:	"/home/owner/programming/c/greetz"
(gdb) x/s 0x080485d8
0x80485d8:	"\033[31;1m"
(gdb) x/s 0x080485e0
0x80485e0:	"\033[0m"
(gdb)


The above relates pretty well with the code we had in C for the printf() call there:

	////////////////////////////////////////////////
	// Escape sequences that will color our output
	////////////////////////////////////////////////
	char *Yellow = "\033[33;1m";
	char *Red = "\033[31;1m";
	char *Normal = "\033[0m";
 
	////////////////////////////////////////////////
	// a printf call with a lot of parameters ;-)
	////////////////////////////////////////////////
	printf("\n\t%s[*] Usage%s: %s %s<Name>%s\n\n", Yellow, Normal, progName, Red, Normal);


gdb: Viewing CPU Registers and Flags

Registers are used as storage by the CPU, on X86 they hold a word as there value. It is possible to view them by using info registers.

(gdb) break printUsage 
Breakpoint 1 at 0x8048491
(gdb) run
Starting program: /home/owner/programming/c/greetz 
 
Breakpoint 1, 0x08048491 in printUsage ()
(gdb) info registers 
eax            0xffffd571	-10895
ecx            0xffffd370	-11408
edx            0xffffd394	-11372
ebx            0xffffd370	-11408
esp            0xffffd320	0xffffd320
ebp            0xffffd338	0xffffd338
esi            0x0	0
edi            0x0	0
eip            0x8048491	0x8048491 <printUsage+6>
eflags         0x282	[ SF IF ]
cs             0x23	35
ss             0x2b	43
ds             0x2b	43
es             0x2b	43
fs             0x0	0
gs             0x63	99
(gdb)

Some of these that are useful in C are the EIP (which points to the current instruction address to be executed.), ESP (Which points to the top of the stack), and EBP (Which points to the bottom of the stack.). EIP can be accessed with $ip. EBP can be accessed with $ebp. Most other registers can be accessed with $[register].


gdb: ~/.gdbinit File

Sometimes you will need to exit and re-enter gdb. If there are a lot of things that need to be setup, you can set them up in this file. This file acts like a script that will get run when you start gdb.


PEDA - The Python Exploit Development Assistant

GDB now has a python API which enabled PEDA to become a useful plugin for GDB. It can be downloaded via https://github.com/longld/peda. While this tool is designed to aid exploit development, it adds a lot of general useful debugging functions to GDB. First is it colors things making it a lot easier on the eyes. It also allows you to run shell commands while in gdb. Also when it breaks it will automatically show you the registers and so the top of the stack and dereference a lot of the values automatically, and let you know if a jump will be followed. Once downloaded you can pull it into gdb by using "source [path/to/peda.py]". Alternatively, this line can be added to your ~/.gdbinit file. To see a full list of functions available. Type "phelp".