## Sunday, June 2, 2013

### printf() debugging in assembly language

I am happy to report that I have figured out a way to do printf() debugging with assembly language!

I know that many people think of printf() debugging as a pedestrian technique, a hack for people who are too lazy or obstinate to use a real debugger. But I've always found it extremely useful, because of the simple fact that stepping through a program is slow. Debugging is all about finding the precise moment where the program's expected behavior diverged from its actual behavior. Stepping through your entire program line-by-line and verifying the expected behavior at every step is slow and thought-intensive, and if you miss the critical moment you have to start over -- debuggers don't go backwards.

(Ed: actually I take this back; it looks like GDB can go backwards these days! I will most definitely have to try this out. But it's still the case that the critical moment may be far behind the moment where the debugger is stopped.)

Contrast this with printf() debugging, which very quickly gives you a transcript of the program's entire execution. This lets you see patterns and hopefully hone in on the crucial moment where things went awry. If you didn't get quite the information you need, you can insert slightly different print statements until the transcript tells you the story.

In pretty much every language, printf() debugging is easy, because you can insert a print statement anywhere and pass it any parameters you want. But when you're hand-writing assembly language, it's a much bigger pain. Making function calls (like to printf()) is far more manual; you have to figure out where to store the string, deal with varargs calling conventions, and even then making the function call will clobber half your registers. It's not very practical.

Looking for a solution to this, I came across the GDB breakpoint command lists. Basically you can give GDB a list of things to do when a breakpoint is hit, like print arbitrary registers, variables, memory, etc! And since the last command can be continue, the program can run without any interactive debugger intervention.

I came across some helpful examples on Stack Overflow (like this one) to give me ideas. So far I've just been using simple commands like:
set width 0
set height 0
set verbose off

b X.0x7ffff5221b34.OP_CHECKDELIM_RET
commands
silent
printf "X.0x7ffff5221b34.OP_CHECKDELIM_RET\n"
continue
end

This will print the name of this function whenever it is hit. I'm using this to debug my current project. I've been working on a bytecode interpreter with a JIT code generator backend. I can already debug the interpreter by making it dump its list of bytecodes as it executes them. With this technique I can debug the JIT in a similar way; I can make GDB trace the execution of my JIT code and dump the bytecode equivalent of what it is doing. That way I can see where my JIT code is diverging from the behavior of my known-good interpreter.

1. Useful commentary on this article that was mailed to me by a person named Icarus:

For this kind of thing breakpoint command lists are fine. The limitation of the GDB scripting is that any kind of error aborts the command list that is being run.

For example, suppose you have a hash table with chaining (i.e. you have an array of structures and each structure has a pointer to another structure) and you want to dump the whole thing. It is essentially 2 nested loops, the outer one to loop over the array and the inner one to walk the linked list until it gets a NULL pointer.

If the data is corrupted (perhaps you freed a structure whilst it was still used by the hash table) then your entire script fails at the point where it de-references the invalid pointer.

The correct fix is to switch to the python facilities that are in newer versions of GDB.

If you don't want to switch, then I suggest the best thing to do is to write your debugging scripts as two separate commands. One command uses global data to tell you a "step", it updates the global data before it attempts to do the step, so if you re-run the command it gives you the next lot of data. The second command just sets up the global data and calls the stepper command.

So for the dump the hash table example, the stepper command would take for example a global that tells it which row to start at. It would update the global so if you rerun the command it starts at the next row. It then would walk the linked list, printing out stuff. If it gets a NULL then it just calls itself to print the next row (in practice you might want to turn this tail recursion into a loop). If it aborts due to corruption you can just run the stepper command again to continue printing as much data as possible.

The "run again" is just press the "Enter" key - see http://sourceware.org/gdb/onlinedocs/gdb/Command-Syntax.html#Command-Syntax

2. These types of tracing breakpoints are great; I use them often. However, they are terribly slow, especially when remote debugging. The problem is further exacerbated by code that is hit often. In these situations, I think other dynamic binary instrumentation techniques are much more efficient.