Baby Steps - Unikraft Internals

We start presenting the modular architecture of Unikraft and how students can build their unikernels using the manual, `make`-based approach. Expected time: 100 minutes

Reminder#

In the previous session, we have experimented with building and running several unikernels using scripts. Today, we will explore the modular architecture of Unikraft, and how we can leverage that in order to setup, configure, build and run our unikernels from scratch, without the use of scripts or pre-defined configurations.

Let's start by understanding the basic layout of the Unikraft working directory, its environment variables, as well as what the most common Unikraft specific files mean. We are also going to take a look at how we can build basic applications and how we can extend their functionality and support by adding ported external libraries.

Before everything, we shall take a bird's eye view of what Unikraft is and what we can do with it. Unikraft is a unikernel SDK, meaning it offers you the blocks (source code, configuration and build system, runtime support) to build and run unikernels (i.e. applications such as redis, nginx or sqlite). A unikernel is a single image file that can be loaded and run as a separate running instance, most often a virtual machine.

Summarily, the Unikraft components are shown in the image below:

unikraft components

Unikraft is the core component, consisting of multiple core / internal libraries (each providing a part of the functionality commonly found in an operating system), the build system, and platform and architecture code. It is the basis of any unikernel image. It is located in the main Unikraft repository.

Libraries are additional software components that will be linked against Unikraft for the final image. There are multiple supported libraries. Each unikernel image is using its specific libraries. Libraries are also called external libraries, as they sit outside the main Unikraft repository. They are typically common libraries (such as OpenSSL - a crypto library, or LwIP - a TCP/IP network stack) that have been ported on top of Unikraft. They are located in specialized repositories in the Unikraft organization, those whose names start with lib-.

Application is the actual application code. It typically provides the main() function (or equivalent) and is reliant on Unikraft and external libraries. Applications that have been ported on top of Unikraft are located in repositories in the Unikraft organization, those whose names start with app-.

An important role of the core Unikraft component is providing support for different platforms and architectures. A platform is the virtualization / runtime environment used to run the resulting image (i.e QEMU, Firecracker, VMWare etc.). An architecture details the CPU and memory specifics that will run the resulting image (i.e. x86, ARM).

We can use a lower-level configuration and build system (based on Kconfig and make) to get a grasp of how everything works. The low-level system will be further detailed in the next session.

Setting up the repositories#

For ease of use, Unikraft applications will follow a predefined folder structure. Namely, the one that is used in the catalog-core repository, that you are already familiar to.

catalog-core/               ---> working directory containing applications
`-- c-hello/                ---> application directory
    |
    |
    |-- hello.c             ---> application code
    |-- Makefile            ---> makefile for specifying external library dependencies
    |-- Config.uk           ---> optional, unikraft configuration options for the application
    |-- Makefile.uk         ---> the application's compiler flags and source files to build
    |
    |
    `-- workdir/
        |-- build/          ---> will be created at build time, all build artifacts will be placed here
        |-- libs/           ---> optional, external libraries used by the application
        `-- unikraft/       ---> unikraft core kernel, containing all the internal libraries
`-- nginx/
    |
    |
    |
`-- redis/
    |
    |
    |
...

This file structure (specifically the workdir/ directory) is created by the setup.sh scripts from the first session, so you should already have it in your catalog-core/ local clone. If not, run the setup.sh scripts again, for every application that you want to build.

Let's begin with the simplest app there is, c-hello.

$ cd catalog-core/c-hello/

$ ls -F
[...]  Makefile  README.md  hello.c  [...]

Now, while inside the c-hello/ directory, we run the setup.sh script to create the workdir/ folder, which will store the Unikraft core kernel from this repository, along with the libs/ directory which should store all of the external libraries the applications depends on, cloned from the lib- repos of the Unikraft organization. Since the c-hello application is fairly simple, it does not use any external library, so there will be no libs/ directory.

Configuration#

First we make sure we clean everything that was configured while running the scripts, so we can start from scratch:

$ make distclean

Now that we have our setup ready, we can start configuring our application!

$ make menuconfig

This will open a menu-driven user interface in which you can choose the target architecture, virtualization platform, and configuration options for both internal and external libraries of the resulting unikernel image.

We are met with the following configuration menu. Let's pick the x86 architecture:

arch selection menu

arch selection menu2

arch selection menu3

Now, press Exit (or hit the Esc key twice) until you return to the initial menu.

We have now set our desired architecture, let's now proceed with the platform. We will choose only the KVM platform (press the Y key to select it), which is supported by QEMU, the program we will use to launch our virtual machine after compilation:

plat selection menu

plat selection menu2

For the moment, this is sufficient. Press the Save button and exit the configuration menu by repeatedly selecting Exit.

Building#

We can now build our application. We can do this by simply invoking make:

$ make -j$(nproc)

This command invokes the build system by compiling the application on $(nproc) different threads, for shorter build times. The compilation process includes all of the Unikraft core internal libraries selected by default for this app, along with the application source files (just main.c for c-hello). The process should end with the following output:

[...]
  LD      c-hello_qemu-x86_64.dbg
  UKBI    c-hello_qemu-x86_64.dbg.bootinfo
  SCSTRIP c-hello_qemu-x86_64
  GZ      c-hello_qemu-x86_64.gz
make[1]: Leaving directory '/projects/unikraft/catalog-core/repos/unikraft'

Notice that each line in make's output consists of an operation in the build process (i.e compiling, linking, symbol striping etc.) and a corresponding generated file.

Running#

Now that the unikernel image has been sucessfully generated, we can finally run it using the QEMU Virtual Machine Monitor: We will always run the stripped image, in our case c-hello_qemu-x86_64.

$ qemu-system-x86_64 -kernel workdir/build/c-hello_qemu-x86_64 -nographic

The -kernel option is used for indicating the image to be booted by the machine, while the -nographic option is simply for using QEMU as a command-line program by redirecting all output to the terminal.

SeaBIOS (version Arch Linux 1.16.2-1-1)


iPXE (http://ipxe.org) 00:03.0 C900 PCI2.10 PnP PMM+06FD3260+06F33260 C900



Booting from ROM..Powered by
o.   .o       _ _               __ _
Oo   Oo  ___ (_) | __ __  __ _ ' _) :_
oO   oO ' _ `| | |/ /  _)' _` | |_|  _)
oOo oOO| | | | |   (| | | (_) |  _) :_
 OoOoO ._, ._:_:_,\_._,  .__,_:_, \___)
                  Atlas 0.13.1~5eb820bd
Hello from Unikraft!
Arguments:  "build/c-hello_qemu-x86_64"

If everything went well, you should see the beautiful Unikraft banner and the "Hello world" message printed out. Congratulations!

Choosing another architecture#

Let's try to build the c-hello app for the ARM64 architecture, as well. All you need to do is head back into the configuration menu by running make menuconfig, after which you should go to the Architecture Selection menu where you can pick the Armv8 compatible (64 bits) target. Make sure to save your changes. When you exit the menu, you will most likely see this warning message:

*** The following configuration changes were detected:
*** - CPU architecture changed
*** - CPU optimization changed
*** Execute 'make clean' before executing 'make'.
*** This is to ensure that the new setting is applied
*** to every compilation unit.

This is a very good recommandation and you'll use it a lot when configuring and building applications - so make sure to run

$ make properclean

...followed by

$ make -j$(nproc)

...to finally build an ARM64 unikernel. Since we are targeting a different architecture, we have to use QEMU to emulate an ARM64 CPU:

$ qemu-system-aarch64 -kernel workdir/build/c-hello_qemu-arm64 -nographic -machine virt -cpu cortex-a57

The -kernel and -nographic are the same as in the x86 case. The -machine option is used to specify which ARM board we want to emulate. We can choose among Raspberry PI, Siemens, NXP boards and so on, but we are instead using the generic platform virt, which does not correspond to real hardware and is the board recommended by QEMU to run guests. Lastly, the -cpu option is used for specifying the CPU whose ISA we want to emulate. Some CPUs differ, for example, with regards to the extensions they implement. The cortex-a57 is sufficient for our examples.

If everything went well, you should be greeted by the same Unikraft banner:

Powered by
o.   .o       _ _               __ _
Oo   Oo  ___ (_) | __ __  __ _ ' _) :_
oO   oO ' _ `| | |/ /  _)' _` | |_|  _)
oOo oOO| | | | |   (| | | (_) |  _) :_
 OoOoO ._, ._:_:_,\_._,  .__,_:_, \___)
                  Atlas 0.13.1~5eb820bd
Hello from Unikraft!

Enabling kernel logs#

We will now try to further configure our c-hello application. For example, we can try to enable debug messages logged by the kernel to gain a better undersanding of the whole booting process. You can target either ARM64, or x86 - it's your choice. We can do this through the same menuconfig interface presented earlier, but first, it is recommended that you clean all of the application's build files in order to make sure that the changes are applied correctly.

$ make properclean

make[1]: Entering directory '/projects/unikraft/catalog-core/repos/unikraft'
  RM      build/
make[1]: Leaving directory '/projects/unikraft/catalog-core/repos/unikraft'

$ make menuconfig

To enable debug messages, we need to configure Unikraft's internal library which handles logging information, ukdebug. To do this, we can proceed to the Library Configuration menu:

library selection menu

We can then see all of Unikraft's internal libraries listed. Scroll down to ukdebug, and hit enter:

ukdebug select1

By default, only error messages are shown. We want to see all the messages (info, warnings, errors) logged by the kernel.

ukdebug select2

ukdebug select3

As before, press the Save button and exit the configuration menu by repeatedly selecting Exit.

Now, if we build and run the application in the same manner (i.e with $ make -j $(nproc), followed by $ qemu-system-x86_64 -kernel build/c-hello_qemu-x86_64 -nographic), we should see a more detailed output:

$ qemu-system-x86_64 -kernel build/c-hello_qemu-x86_64 -nographic

SeaBIOS (version Arch Linux 1.16.2-1-1)


iPXE (http://ipxe.org) 00:03.0 C900 PCI2.10 PnP PMM+06FD3260+06F33260 C900



Booting from ROM..[    0.000000] Info: [libkvmplat] <setup.c @  245> Memory 00fd00000000-010000000000 outside mapped area
[    0.000000] Info: [libukconsole] <console.c @  176> Registered con1: vgacons, flags: -O
[    0.000000] Info: [libkvmplat] <memory.c @  498> Memory 00fd00000000-010000000000 outside mapped area
[    0.000000] Info: [libkvmplat] <setup.c @   99> Switch from bootstrap stack to stack @0x11000
[    0.000000] Info: [libukboot] <boot.c @  280> Unikraft constructor table at 0x138000 - 0x138018
[    0.000000] Info: [libukboot] <boot.c @  289> Initialize memory allocator...
[    0.000000] Info: [libukallocbbuddy] <bbuddy.c @  584> Initialize binary buddy allocator 11000
[    0.000000] Info: [libukboot] <boot.c @  348> Initialize the IRQ subsystem...
[    0.000000] Info: [libukboot] <boot.c @  355> Initialize platform time...
[    0.000000] Info: [libkvmplat] <tscclock.c @  255> Calibrating TSC clock against i8254 timer
[    0.100092] Info: [libkvmplat] <tscclock.c @  276> Clock source: TSC, frequency estimate is 2371711100 Hz
[    0.100678] Info: [libukboot] <boot.c @  359> Initialize scheduling...
[    0.101083] Info: [libukschedcoop] <schedcoop.c @  289> Initializing cooperative scheduler
[    0.103501] Info: [libukboot] <boot.c @  392> Init Table @ 0x138018 - 0x138038
[    0.104000] Info: [libukbus] <bus.c @  133> Initialize bus handlers...
[    0.104424] Info: [libukbus] <bus.c @  135> Probe buses...
Powered by
o.   .o       _ _               __ _
Oo   Oo  ___ (_) | __ __  __ _ ' _) :_
oO   oO ' _ `| | |/ /  _)' _` | |_|  _)
oOo oOO| | | | |   (| | | (_) |  _) :_
 OoOoO ._, ._:_:_,\_._,  .__,_:_, \___)
Pan 0.19.0~9603a4ab
[    0.110409] Info: [libukboot] <boot.c @  472> Pre-init table at 0x138080 - 0x138080
[    0.111628] Info: [libukboot] <boot.c @  483> Constructor table at 0x138080 - 0x138080
[    0.112924] Info: [libukboot] <boot.c @  506> Calling main(1, ['workdir/build/c-hello_qemu-x86_64'])
Hello from Unikraft!
[    0.115008] Info: [libukboot] <boot.c @  516> main returned 0
[    0.115729] Info: [libukboot] <boot.c @  432> Halting system (0)
[    0.116478] Info: [libkvmplat] <shutdown.c @   59> Unikraft halted

You can notice that the debug logs prove to be quite useful: they include information such as timestamp, debugging level (warning, info, error etc.), internal library component (libkvmpci, libukschedcoop etc.) and the source file along with the line in it.

Enabling tests#

As an exercise, try to enable the uktest internal library from the configuration menu. This should trigger the execution of a suite of tests for some of Unikraft's internal libraries during the booting of the unikernel image. You should get an output similar to this one:

$ qemu-system-x86_64 -kernel build/c-hello_qemu-x86_64 -nographic

SeaBIOS (version Arch Linux 1.16.2-1-1)


iPXE (http://ipxe.org) 00:03.0 C900 PCI2.10 PnP PMM+06FD3260+06F33260 C900


[...]


[    0.000000] Info: test: uktest_myself_testsuite->uktest_test_sanity
[    0.000000] Info:    expected `((void *) 0)` to be 0 and was 0 ....................................... [ PASSED ]
[    0.000000] Info:    expected `1` to not be 0 and was 0x1 ............................................ [ PASSED ]
[    0.000000] Info:    expected `0` to be 0 and was 0 .................................................. [ PASSED ]
[    0.000000] Info:    expected `1` to not be 0 and was 1 .............................................. [ PASSED ]
[    0.000000] Info:    expected `a` to be 0x182f3c and was 0x182f3c .................................... [ PASSED ]
[    0.000000] Info:    expected `a` at 0x182f3c to equal `b` at 0x182f3c ............................... [ PASSED ]
[    0.000000] Info:    expected `1` to be 1 and was 1 .................................................. [ PASSED ]
[    0.000000] Info:    expected `0` to not be 1 and was 0 .............................................. [ PASSED ]
[    0.000000] Info:    expected `1` to be greater than 0 and was 1 ..................................... [ PASSED ]
[    0.000000] Info:    expected `1` to be greater than or equal to 1 and was 1 ......................... [ PASSED ]
[    0.000000] Info:    expected `2` to be greater than or equal to 1 and was 2 ......................... [ PASSED ]
[    0.000000] Info:    expected `0` to be less than 1 and was 0 ........................................ [ PASSED ]
[    0.000000] Info:    expected `1` to be less than or equal to 1 and was 1 ............................ [ PASSED ]
[    0.000000] Info:    expected `1` to be less than or equal to 2 and was 1 ............................ [ PASSED ]
[    0.000000] Info: [libukboot] <boot.c @  288> Initialize memory allocator...


[...]

[    0.138087] Info: [libuktest] <test.c @   80> uktest:suites:     2 total
[    0.138689] Info: [libuktest] <test.c @   93> uktest:cases:      6 total
[    0.139226] Info: [libuktest] <test.c @  106> uktest:assertions: 24 total
Powered by
o.   .o       _ _               __ _
Oo   Oo  ___ (_) | __ __  __ _ ' _) :_
oO   oO ' _ `| | |/ /  _)' _` | |_|  _)
oOo oOO| | | | |   (| | | (_) |  _) :_
 OoOoO ._, ._:_:_,\_._,  .__,_:_, \___)
                  Atlas 0.13.1~5eb820bd
[    0.142533] Info: [libukboot] <boot.c @  369> Pre-init table at 0x1220a8 - 0x1220a8
[    0.143355] Info: [libukboot] <boot.c @  380> Constructor table at 0x1220a8 - 0x1220a8
[    0.144165] Info: [libukboot] <boot.c @  401> Calling main(1, ['build/c-hello_qemu-x86_64'])
Hello from Unikraft!
Arguments:  "build/c-hello_qemu-x86_64"
[    0.145824] Info: [libukboot] <boot.c @  410> main returned 0, halting system
[    0.146611] Info: [libkvmplat] <shutdown.c @   35> Unikraft halted

Tasks#

After you toy around with the c-hello application, move to a more complex application that will require external libraries (e.g c-http). See the differences in the Makefiles, in the workdir/ directory and in the config Library menu. Try to make the c-http application build without errors. As a reference, you can run the build script from the first session, then run make menuconfig and see what the build script selected. Also look at the c-http run script to see how you can add networking to qemu.

After you sucessfully build and run c-http manually, move to a more complex example, nginx. Nginx will also require a filesystem, check the build and run scripts to see how that works.