Debugging

Unikernels aim to be a more efficient method of virtualization, and this can sometimes cause problems. This guide aims to familiarize you with solving problems encountered during the development using GDB. This is going to be a more practical guide, with a focus on exercises.

Reminders#

At this stage, you should be familiar with the steps of configuring, building and running any application within Unikraft and know the main parts of the architecture. Below you can see a list of useful commands.

Today we'll make use of gdb. We recommend the following cheat sheet for the most common commands. A quick crash course on GDB may be found here.

Debugging#

Contrary to popular belief, debugging an unikernel is simpler than debugging a standard operating system. Since the application and OS are linked into a single binary, debuggers can be used on the running unikernel to debug both application and OS code at the same time. A couple of things you should know before you get started:

1. In the configuration menu (presented with make menuconfig), under Build Options make sure that Drop unused functions and data is unselected. This prevents Unikraft from removing unused symbols from the final image and, if enabled, might hide missing dependencies during development.
2. Use make V=1 to see verbose output for all the commands being executed during the build. If the compilation for a particular file is breaking and you would like to understand why (e.g., perhaps the included paths are wrong), you can debug things by adding the -E flag to the command, removing the -o [objname], and redirecting the output to a file which you can then inspect.
3. Check out the targets under Miscellaneous when typing make help, these may come in handy. For instance, make print-vars enables inspecting at the value of a particular variable in Makefile.uk.

Using GDB#

The build system always creates two image files for each selected platform:

• one that includes debugging information and symbols (.dbg file extension)
• one that does not

Before using GDB, make sure compiler optimization is turned off: in the configuration menu, go to Build Options --> Optimization level and select No optimizations. Then, under Build Options, select a Debug information level that is bigger than 0. We recommend 3, the highest level.

Selecting the Debug Level

Once set, save the configuration and build your images.

KVM/Qemu#

You must run the image that does not contain debugging information (as in no .dbg extension). We will use the .dbg image with gdb. We recommend creating the guest in a paused state (the -S option):

qemu-system-x86_64 -s -S -nographic -kernel workdir/build/hello-c_qemu-x86_64

Note that the -s parameter is shorthand for -gdb tcp::1234.

Now connect GDB by using the debug image with:

gdb --eval-command="target remote :1234" workdir/build/app-helloworld_kvm-x86_64.dbg

Note: you will want to run gdb using the .dbg command, otherwise you will not have any debugging symbols.

Unless you're debugging early boot code, you’ll need to set a hardware break point. Hardware breakpoints have the same effect as the common software breakpoints you are used to, but they are different in the implementation. As the name suggests, hardware breakpoints are based on direct hardware support. This may limit the number of breakpoints you can set, but makes them especially useful when debugging kernel code.

hbreak [location]
continue

You can now run continue and debug as you would do normally.

Work Items#

01. Tutorial: Use GDB in Unikraft#

Build the c-hello application, like you did in the last session. Enable build messages, attach gdb to the application and toy around. Break at the main function, see if you can make the application print a different message without changing the source code.

02. Mystery: Find the secret using GDB#

Before starting the task, let's get familiar with some GDB commands.

ni - go to the next instruction, but skip function calls

si - go to the next instruction, but enters function calls

c - continue execution to the next breakpoint

p expr - display the value of an expression

x addr - get the value at the indicated address (similar to p *addr)

whatis arg - print the data type of arg

GDB provides convenience variables that you can use within GDB to hold on to a value and refer to it later. For example:

(gdb) set $foo = *object_ptr

Note that you can also cast variables in GDB similar to C:

(gdb) set $var = (int *) ptr

If you want to dereference a pointer and actually see the value, you can use the following command:

(gdb) p *addr

You can find more GDB commands here Also, if you are unfamiliar with X86_64 calling convention you can read more about it here.

Now, let's get back to the task. Download the target kernel image from here.

Use qemu and gdb to navigate through the mystery_kvm-x86_64.dbg file.

Run the application using the following command:

qemu-system-x86_64 -nographic -kernel mystery_kvm-x86_64.dbg -s -S

You can attach to it from another terminal using

gdb --eval-command="target remote :1234" mystery_kvm-x86_64.dbg

Do you think you can find out the secret?

1. Look through the assembly code and see what it does. Follow the test eax, eax instructions and see what needs to happen to pass those tests and get to the last point where the secret is printed (i.e. be able to advance to address 0x000000000018a4ae).

2. Investigate memory addresses (using the x instruction - such as x/s $rbp-0x120), do instruction stepping (stepi or nexti), use breakpoints (break *<address>) and find out the secret.

03. Nginx with or without main? That's the question#

Let's try a new application based on networking, Nginx.

Use the same setup from the last session, follow the instructions in the README.md file from the catalog repository and build Nginx for Qemu/KVM. Do you observe something strange? Where is the main.c file?

Edit the Config.uk file and remove the select LIBNGINX_MAIN_FUNCTION line. Deselect this make menuconfig option Library Configuration -> libnginx -> Provide a main function and try to build and run the Nginx application again. Try to create your own main.c file so that the application works again.

In order to add your main file to the build system, you must add it to the Makefile.uk file. Check the c-hello Makefile.uk to see an example.

Basically, this exercise has two tasks:

• Build Nginx, as in the previous session
• Nginx without provide main function

Hints:

• Use nm on the kernel image to see what functions are available.
• Check the files in workdir/libs/nginx.

Further Reading#

• Hardware Breakpoint
• Tracepoints
• GDB Cheatsheet
• X86_64 calling convention