Overview

In this session we take a look of what’s happening behind the scenes when building and running Unikraft. We take a dive into the internals of the build system and how Unikraft works with different applications. Each application requires a specific configuration that specializes the Unikraft image to its requirements.

01. Unikraft Core

The Unikraft core is comprised of several components:

• the architecture code: This defines behaviours and hardware interactions specific to the target architecture (x86_64, ARM, RISC-V). For example, for the x86_64 architecture, this component defines the usable registers, data types sizes and how Thread-Local Storage should happen.
• the platform code: This defines interaction with the underlying hardware, depending on whether a hypervisor is present or not, and which hypervisor is present. For example, if the KVM hypervisor is present, Unikraft will behave almost as if it runs bare-metal, needing to initialize the hardware components according to the manufacturer specifications. The difference from bare-metal is made only at the entry, where some information, like the memory layout, the available console, are supplied by the bootloader (Multiboot), and there’s no need to interact with the BIOS or UEFI. In the case of Xen, many of the hardware-related operations must be done through hypercalls, thus reducing the direct interaction of Unikraft with the hardware.
• internal libraries: These define behaviour independent of the hardware, like scheduling, networking, memory allocation, basic file systems. These libraries are the same for every platform or architecture, and rely on the platform code and the architecture code to perform the needed actions. The internal libraries differ from the external ones in the implemented functionalities. The internal ones define parts of the kernel, while the external ones define user-space level functionalities. For example, uknetdev and lwip are 2 libraries that define networking components. Uknetdev is an internal library that interacts with the network card and defines how packages are sent using it. Lwip is an external library that defines networking protocols, like IP, TCP, UDP. This library knows that the packages are somehow sent over the NIC, but it is not concerned how. That is the job of the kernel.

02. Configuring Unikraft

Unikraft is a configurable operating system, where each component can be modified / configured, according to the user’s needs. This configuration is done using a version of Kconfig (used in the Linux kernel), through the Config.uk files. In these files, options are added to enable libraries, applications and different components of the Unikraft core. The user can then apply those configuration options, using make menuconfig, which generates an internal configuration file, .config, that can be understood by the build system. Once configured, the Unikraft image can be built, using make, and run, using the appropriate method (Linux ELF loader, qemu-kvm, xen, others).

Configuration can be done in a few ways:

• Manually, using

  $ make menuconfig
  

• Adding a dependency in Config.uk for a component, so that the dependency gets automatically selected when the component is enabled. This is done using the depends on and select keywords in Config.uk. The configuration gets loaded and the .config file is generated by running

  $ make menuconfig
  

  This type of configuration removes some configuration steps, but not all of them.

In this session, we will use the first configuration options.

03. Building Unikraft

Once the application is configured, in .config, symbols are defined (e.g. CONFIG_ARCH_X86_64). Those symbols are usable both in the C code, to include certain functionalities only if they were selected in the configuring process, and in the actual building process, to include / exclude source files, or whole libraries. This last step is done in Makefile.uk, where source code files are added to libraries. During the build process, all the Makefile.uk files (from the Unikraft core and external libraries) are evaluated, and the selected files are compiled and linked, to form the Unikraft image.

04. Running Unikraft

Summary

• Unikraft is a special type of operating system, that can be configured to match the needs of a specific application.
• This configuration is made possible by a system based on Kconfig, that uses Config.uk files to add possible configurations, and .config files to store the specific configuration for a build.
• The configuration step creates symbols that are visible in both Makefiles and source code.
• Each component has its own Makefile.uk, where source files can be added, removed, or be made dependent on the configuration.
• Unikraft has an internal libc, but it can use others, more complex and complete, like musl.
• Being an operating system, it needs to be run by a hypervisor, like KVM or xen, to work at full capacity. It can also be run as an ELF, in Linux, but in this way the true power of Unikraft is not achieved.

Work Items

Support Files

Session support files are available in the repository. If you already cloned the repository, update it and enter the session directory:

$ cd path/to/repository/clone

$ git pull --rebase

$ cd content/en/community/hackathons/sessions/behind-scenes/

$ ls
demo  images  index.md  sol  work

If you haven’t cloned the repository yet, clone it and enter the session directory:

$ git clone https://github.com/unikraft/docs.git

$ cd docs/

$ cd content/en/community/hackathons/sessions/behind-scenes/

$ ls
demo  images  index.md  sol  work

01. Tutorial: Building and Running Unikraft helloworld

We want to build the helloworld application, using the Kconfig-based system.

Set Up

It is recommended that for building and developing applications and Unikraft, you create a conventional folder structure:

.
|-- apps/
|-- libs/
`-- unikraft/

That is a hierarchy with:

• unikraft/ as the clone of the unikraft repository
• apps/ storing folders for applications
• libs/ storing folders for libraries

You would usually only have a single such hierarchy and add applications and / or libraries in their respective folders and use a single clone of the unikraft repository. We create this hierarchy, if not having it created already, by using the commands:

$ mkdir workdir

$ cd workdir/

$ mkdir apps libs

$ git clone https://github.com/unikraft/unikraft
[...]

$ tree --charset=ascii -L 1
.
|-- apps
|-- libs
`-- unikraft

We want to work on the helloworld application, so we clone it in the apps/ subfolder:

$ cd apps/
$ git clone https://github.com/unikraft/app-helloworld helloworld

In the apps/helloworld/ folder, make sure that UK_ROOT and UK_LIBS are set correctly in the Makefile file, i.e. to point to the location of the unikraft repository clone, and to the folder storing library repositories. If you are not sure if they are set correctly, set them like this:

UK_ROOT ?= $(PWD)/../../unikraft
UK_LIBS ?= $(PWD)/../../libs

KVM, x86_64

We build the Unikraft helloworld image for the kvm platform, for the x86_64 architecture. We follow the steps below.

Clean the Environment

That means remove build files (the build/ directory) and configuration files (the .config file).

$ make distclean

While this may not always be required, it’s the safest option to make sure that previous build / configuration artifacts are removed. Previous build / configuration artifacts may give out configuration and build errors.

If you only wanted to remove the build files (and not the .config file), you would use:

$ make properclean

In short, make distclean removes everything, make properclean keeps the .config file.

Configure the Application

To enter the configuration screen, run:

$ make menuconfig

In the configuration screen, follow the steps:

1. From Architecture Selection, select Architecture -> x86 compatible.
2. From Platform Configuration, select KVM guest.
3. Save and exit.
4. You can now check that the .config file is created.

Build the Application

Build the application by running

$ make prepare
$ make -j $(nproc)

Run the Unikraft Application

Run the resulting image with QEMU:

$ sudo qemu-system-x86_64 -kernel ./build/app-helloworld_kvm-x86_64 -nographic

Besides -nographic, no other option is needed to run the app-helloworld application. Other, more complex applications, will require more options given to the qemu-system-x86_64 command.

We have run Unikraft in the emulation mode, with the command from above. This is the compatible way of running Unikraft, that works on any Linux-based setup, including inside a virtual machine.

However, for actual performance, we want to run it in the virtualization mode, by adding the -enable-kvm option. Note that this requires KVM support both in your Linux-based setup.

After adding the -enable-kvm option to the command above, you may receive a warning: host doesn't support requested feature:. This is because KVM uses a generic CPU model. You can instruct KVM to use your local CPU model, by adding -cpu host to the command.

The final command will look like this:

$ sudo qemu-system-x86_64 -enable-kvm -cpu host -kernel ./build/app-helloworld_kvm-x86_64 -nographic

While we are here, we can check some differences between emulation and virtualization. Record the time needed by each image to run, using time, like this:

$ time sudo qemu-system-x86_64 -kernel ./build/app-helloworld_kvm-x86_64 -nographic
$ time sudo qemu-system-x86_64 -enable-kvm -cpu host -kernel ./build/app-helloworld_kvm-x86_64 -nographic

Because helloworld is a simple application, the real running time will be similar. The differences are where each image runs most of its time: in user space, or in kernel space. As a task for you, find an explanation to the differences.

02. Tutorial: Make It Speak

KVM, x86_64

We want to enable debugging for the KVM platform and the x86_64 architecture. We do this by enabling the ukdebug library in the configuration menu. It is located in the Library Configuration menu.

We follow the steps.

Clean the Environment

$ make distclean

Configure the Application

$ make menuconfig

In the configuration screen, follow the steps:

1. From Architecture Selection, select Architecture -> x86 compatible.
2. From Platform Configuration, select KVM guest.
3. From Library Configuration, enter the ukdebug menu.
   1. Select the Enable kernel messages (uk_printk) entry.
   2. Change the option below it, Kernel message level, from Show critical and error messages (default) to Show all types of messages.
   3. To make things prettier, also enable the Colored output option.
4. Save and exit.

Build the Application

$ make prepare
$ make -j $(nproc)

Run the Unikraft Application

$ sudo qemu-system-x86_64 -kernel ./build/app-helloworld_kvm-x86_64 -nographic

We see what timer is used, the i8254 one. Also, we see that the PCI bus is used. And other boot-related items.

03. Tutorial: Adding Filesystems to an Application

For this tutorial, the aim is to create a simple QEMU / KVM application that reads from a file and displays the contents to standard output. A local directory is to be mounted as the root directory (/) inside the QEMU / KVM virtual machine.

We will use both the manual approach (make and qemu-system-x86_64 / qemu-guest) and kraft to configure, build and run the application.

Setup

The basic setup is in the work/06-adding-filesystems/ folder in the session directory. Enter that folder:

$ cd work/06-adding-filesystems/

$ ls -F
guest_fs/  kraft.yaml  launch.sh*  main.c  Makefile  Makefile.uk  qemu-guest*

The guest_fs/ local directory is to be mounted as the root directory (/) inside the QEMU / KVM virtual machine. It contains the grass file. The program (main.c) reads the contents of the /grass file and prints it to standard output. Makefile.uk lists the main.c file as the application source file to be compiled and linked with Unikraft.

Makefile is used by the manual configuration and build system. kraft.yaml is used by kraft to configure, build and run the application.

launch.sh is a wrapper script around qemu-system-x86_64 used to manually run the application. Similarly, qemu-guest is a wrapper script used internally by kraft. We’ll use it as well to run the application.

Important: This setup belongs as an application folder in the apps/ folder in your working directory as discussed in the 1st tutorial of this session. Your best approach would be to copy this folder (work/06-adding-filesystems/) to the apps/ folder in your working directory. You will then get a hierarchy such as:

.
|-- apps/
|   |-- 06-adding-filesystems/
|   `-- helloworld/
|-- libs/
`-- unikraft/

If, at any point of this tutorial, something doesn’t work, or you want a quick check, see the reference solution in sol/06-adding-filesystems/ folder in the session directory.

Using the Manual Approach

Firstly, we will use the manual approach to configure, build and run the application.

Configure

For filesystem functionalities (opening, reading, writing files) we require a more powerful libc. Musl is already ported in Unikraft and will do nicely. For this, we update the LIBS line in the Makefile:

LIBS := $(UK_LIBS)/lib-musl

Update the UK_ROOT and UK_LIBS variables in the Makefile to point to the folders storing the Unikraft and libraries repositories.

Make sure that both unikraft and musl repositories are on the staging branch. Go to each of the two repository folders (unikraft and musl) and check the current branch:

$ git checkout

Now we need to enable 9pfs and musl in Unikraft. To do this, we run:

$ make menuconfig

We need to select the following options, from the Library Configuration menu:

• libmusl
• vfscore: VFS Core Interface
• vfscore: VFS Configuration -> Automatically mount a root filesystem -> Default root filesystem -> 9pfs
  • For the Default root device option fill the fs0 string (instead of the default rootfs string).

These configurations will also mark as required 9pfs and uk9p in the menu.

We want to run Unikraft with QEMU / KVM, so we must select KVM guest in the Platform Configuration menu. For 9pfs we also need to enable, in the KVM guest options menu, Virtio -> Virtio PCI device support.

Save the configuration and exit.

Do a quick check of the configuration in .config by pitting it against the config.sol file in the reference solution:

$ diff -u .config ../../sol/06-adding-filesytstems/config.sol

Differences should be minimal, such as the application identifier.

Build

Build the Unikraft image:

make

Building the Unikraft image will take a while. It has to pull musl source code, patch it and then build it, together with the Unikraft source code.

Run with qemu-system-x86_64

To run the Unikraft image with QEMU / KVM, we use the wrapper launch.sh script, that calls qemu-system-x86_64 command with the proper arguments:

$ ./launch.sh ./build/unikraft-kraft-9pfs-issue_kvm-x86_64
[...]
o.   .o       _ _               __ _
Oo   Oo  ___ (_) | __ __  __ _ ' _) :_
oO   oO ' _ `| | |/ /  _)' _` | |_|  _)
oOo oOO| | | | |   (| | | (_) |  _) :_
 OoOoO ._, ._:_:_,\_._,  .__,_:_, \___)
                   Tethys 0.5.0~825b115
Hello, world!
File contents: The grass is green!
Bye, world!

A completely manual run would use the command:

$ qemu-system-x86_64 -fsdev local,id=myid,path=guest_fs,security_model=none -device virtio-9p-pci,fsdev=myid,mount_tag=fs0 -kernel build/06-adding-filesystems_kvm-x86_64 -nographic
[...]
Powered by
o.   .o       _ _               __ _
Oo   Oo  ___ (_) | __ __  __ _ ' _) :_
oO   oO ' _ `| | |/ /  _)' _` | |_|  _)
oOo oOO| | | | |   (| | | (_) |  _) :_
 OoOoO ._, ._:_:_,\_._,  .__,_:_, \___)
                   Tethys 0.5.0~825b115
Hello, world!
File contents: The grass is green!
Bye, world!

Lets break it down:

• -fsdev local,id=myid,path=guest_fs,security_model=none - assign an id (myid) to the guest_fs/ local folder
• -device virtio-9p-pci,fsdev=myid,mount_tag=fs0 - create a device with the 9pfs type, assign the myid for the -fsdev option and also assign the mount tag that we configured above (fs0) Unikraft will look after that mount tag when trying to mount the filesystem, so it is important that the mount tag from the configuration is the same as the one given as argument to qemu.
• -kernel build/06-adding-filesystems_kvm-x86_64 - tells QEMU that it will run a kernel; if this parameter is omitted, QEMU will think it runs a raw file
• -nographic - prints the output of QEMU to the standard output, it doesn’t open a graphical window

Run with qemu-guest

qemu-guest is the script used by kraft to run its QEMU / KVM images. Before looking at the command, take some time to look through the script, and maybe figure out the arguments needed for our task.

To run a QEMU / KVM application using qemu-guest, we use:

$ ./qemu-guest -e guest_fs/ -k build/06-adding-filesystems_kvm-x86_64

If we add the -D option, we can see the qemu-system command generated.

You may get the following error:

[    0.100664] CRIT: [libvfscore] <rootfs.c @  122> Failed to mount /: 22

If you do, check that the mount tag in the configuration is the same as the one used by qemu-guest. qemu-guest will use the tag fs0.

The fs0 tag is hardcoded for qemu-guest (and, thus, for kraft). This is why we used the fs0 tag when configuring the application with make menuconfig. Another tag could be used but then we couldn’t run the application with qemu-guest or kraft. It could only be run by manually using qemu-system-x86_64 with the corresponding arguments.