Unikraft
Automatically deploy lightweight VMs with 20-50% savings with Unikraft Cloud.

Overview

Debugging

Contrary to popular belief, debugging a unikernel is in fact simpler than debugging a standard operating system. Since the application and OS are linked into a single binary, debuggers can be used on the running unikernel to debug both application and OS code at the same time. A couple of hints that should help starting:

  1. In the configuration menu (presented with make menuconfig), under Build Options make sure that Drop unused functions and data is unselected. This prevents Unikraft from removing unused symbols from the final image and, if enabled, might hide missing dependencies during development.
  2. Use make V=1 to see verbose output for all of the commands being executed during the build. If the compilation for a particular file is breaking and you would like to understand why (e.g., perhaps the include paths are wrong), you can debug things by adding the -E flag to the command, removing the -o [objname], and redirecting the output to a file which you can then inspect.
  3. Check out the targets under Miscellaneous when typing make help, these may come in handy. For instance, make print-vars enables inspecting at the value of a particular variable in Makefile.uk.
  4. Use the individual make clean-[libname] targets to ensure that you’re cleaning only the part of Unikraft you’re working on and not all the libraries that it may depend on. This will speed up the build and thus the development process.
  5. Use the Linux user space platform target (linuxu) for quicker and easier development and debugging.

Using GDB

The build system always creates two image files for each selected platform:

  • one that includes debugging information and symbols (.dbg file extension)
  • one that does not

Before using GDB, go to the configuration menu under Build Options and select a Debug information level that is bigger than 0. We recommend 3, the highest level.

debug information level

Once set, save the configuration and build your images.

Linuxu

For the Linux user space target (linuxu) simply point GDB to the resulting debug image, for example:

$ gdb build/app-helloworld_linuxu-x86_64.dbg

KVM

For KVM, you can start the guest with the kernel image that includes debugging information, or the one that does not. We recommend creating the guest in a paused state (the -S option):

$ qemu-system-x86_64 -s -S -cpu host -enable-kvm -m 128 -nodefaults -no-acpi -display none -nographic -device isa-debug-exit -kernel build/app-helloworld_kvm-x86_64.dbg -append verbose

Note that the -s parameter is shorthand for -gdb tcp::1234. To avoid this long qemu-system-x86 command with a lot of arguments, we can use qemu-guest.

$ qemu-guest -P -g 1234 -k build/app-helloworld_kvm-x86_64.dbg

Now connect GDB by using the debug image with:

$ gdb --eval-command="target remote :1234" build/app-helloworld_kvm-x86_64.dbg

Unless you’re debugging early boot code (until _libkvmplat_start32), you’ll need to set a hardware break point. Hardware breakpoints have the same effect as the common software breakpoints you are used to, but they are different in the implementation. As the name suggests, hardware breakpoints are based on direct hardware support. This may limit the number of breakpoints you can set, but makes them especially useful when debugging kernel code.

hbreak [location]
continue

We’ll now need to set the right CPU architecture:

disconnect
set arch i386:x86-64:intel

And reconnect:

tar remote localhost:1234

You can now run continue and debug as you would do normally.

Xen

For Xen, you first need to create a VM configuration (save it under helloworld.cfg):

name          = 'helloworld'
vcpus         = '1'
memory        = '4'
kernel        = 'build/app-helloworld_xen-x86_64.dbg'

Start the virtual machine with:

$ xl create -c helloworld.cfg

For Xen the process is slightly more complicated and depends on Xen’s gdbsx tool. First you’ll need to make sure you have the tool on your system. Here are sample instructions to do that:

[get Xen sources]
$ ./configure
$ cd tools/debugger/gdbsx/ && make

The gdbsx tool will then be under tools/debugger. For the actual debugging, you first need to create the guest (we recommend paused state: xl create -p), note its domain ID (xl list) and execute the debugger backend:

$ gdbsx -a [DOMAIN ID] 64 [PORT]

You can then connect GDB within a separate console and you’re ready to debug:

$ gdb --eval-command="target remote :[PORT]" build/helloworld_xen-x86_64.dbg

You should also be able to use the debugging file (build/app-helloworld_xen-x86_64.dbg) for GDB instead passing the kernel image.

Practical Work

Support Files

Session support files are available in the repository. If you already cloned the repository, update it and enter the session directory:

$ cd path/to/repository/clone

$ git pull --rebase

$ cd content/en/community/hackathons/sessions/debugging

$ ls
content  demo  images  index.md  sol  work

If you haven’t cloned the repository yet, clone it and enter the session directory:

$ git clone https://github.com/unikraft/docs.git

$ cd content/en/community/hackathons/sessions/debugging

$ ls
content  demo  images  index.md  sol  work

01. Tutorial. Use GDB in Unikraft

For this tutorial, we will just start the app-helloworld application and inspect it with the help of GDB.

First make sure you have the following conventional working directory also shown in Section 02: Behind the Scenes.

.
|-- apps/
|   `-- helloworld/
|-- libs/
`-- unikraft/

For instructions on building app-hellworld using the manual method, see the application README or Section 02: Behind the Scenes.

Linuxu

For the image for the linuxu platform we can use GDB directly with the binary already created.

$ gdb build/app-helloworld_linuxu-x86_64.dbg

KVM

To avoid using a command with a lot of parameters that you noticed above in the KVM section, we can use the qemu-guest script from kraft.

$ wget https://raw.githubusercontent.com/unikraft/kraft/staging/scripts/qemu-guest

$ chmod a+x qemu-guest

$ ./qemu-guest -P -g 1234 -k build/app-helloworld_kvm-x86_64.dbg

Open another terminal to connect to GDB by using the debug image with:

$ gdb --eval-command="target remote :1234" build/app-helloworld_kvm-x86_64.dbg

First you can set the right CPU architecture and then reconnect:

disconnect
set arch i386:x86-64:intel
tar remote localhost:1234

Then you can put a hardware break point at main function and run continue:

hbreak main
continue

All steps described above can be done using the script kvm_gdb_debug located in the work/01-tutorial-gdb/ folder. All you need to do is to provide the path to kernel image.

kvm_gdb_debug build/app-helloworld_kvm-x86_64.dbg

02. Bug or feature?

There are two kernel images located in the work/03-app-bug/ folder. One of them is build for Linuxu, the other for KVM.

First, try to inspect what is wrong with the Linuxu image. You will notice that if you run the program you will get a segmentation fault. Why does this happen?

After you figure out what is happening with the Linuxu image, have a look at the KVM one. It was built from the code source, but when you will try to run it, you will not get a segmentation fault. Is this a bug or a feature?

Support Instructions

Use the connect.sh and debug.sh scripts located in the task directory for debugging a Unikraft instance.

Follow these steps:

  1. Check the disassembly code of main() both in the Linuxu and the KVM image. Use GDB and then use the commands:

    hbreak main

    to break at the main() function. The use

    c

    (for continue) to get the to main() function.

    set disassembly-flavor intel
disass

  2. Use nexti and stepi instructions to step through the code. Get a general idea of what the program does.

  3. Check the values of the rax, rdx and rbp registers. Use

    info registers

    for that.

  4. Inspect the value of registers right after the segmentation fault message.

  5. Deduce what the issue is.

Further Reading