The focus of this session will be on porting new libraries to Unikraft and preparing them for upstreaming to the main organization’s GitHub.

Being a library operating system, the unikernels created using Unikraft are mainly a collection of internal and external libraries, alongside the ported application. As a consequence, a large library pool is mandatory in order to make this project compatible with as many applications as possible.

Reminders

From earlier sessions, we saw that we can add an external library as a dependency for an application by appending it to the $LIBS variable in the application’s Makefile:

LIBS := $(UK_LIBS)/my_lib

Having done that, we can then select it in the menuconfig interface in order for the library to be included in the build process.

Running an unikernel built for kvm can be done using the qemu command as follows:

$ qemu-system-x86_64 -kernel unikraft_unikernel -nographic

The -nographic argument redirects the output generated by the unikernel to the console.

In Session 02: Behind the Scenes we saw that there are two types of libraries:

• internal, which are generally part of the kernel / core (schedulers, file systems, etc.);
• external: which mostly provide user space-level functionalities

The external libraries should be placed in the $UK_LIBS folder, which is by default $UK_WORKDIR/libs, and the applications should be placed in the $UK_APPS folder, which is by default $UK_WORKDIR/apps.

The default working directory structure looks like this:

|-- apps - This is where you would normally place existing app builds
|-- archs - Here we place our custom arch's files
|-- libs - This is where the build system looks for external library pool sources
|-- plats - The files for our custom plats are placed here
`-- unikraft - The core source code of the Unikraft Unikernel

and the relevant environment variables regarding the working directories are:

Overview

Support Files

Session support files are available in the repository. If you already cloned the repository, update it and enter the session directory:

$ cd path/to/repository/clone

$ git pull --rebase

$ cd content/en/community/hackathons/sessions/contributing-to-unikraft/

$ ls -F
index.md  work/ sol/ content/

If you haven’t cloned the repository yet, clone it and enter the session directory:

$ git clone https://github.com/unikraft/docs

$ cd docs/

$ cd docs/content/en/community/hackathons/sessions/contributing-to-unikraft/

$ ls -F
index.md  work/ sol/ content/

Git Structure

The organization’s GitHub contains the main Unikraft repository and separate repositories for external libraries, as well as already ported apps. In the previous sessions, we saw that the Unikraft repository consists of internal libraries, platform code and architecture code. The Unikraft code doesn’t have any external dependencies, in contrast to the external libraries or applications, which can have external dependencies.

External libraries can have more specific purposes. So, we can port a library even just for a single application. The process of adding new internal libraries is almost the same as for external ones, so further we will focus on porting an external library.

Also, the main repository has open issues to which you can contribute. In general, this process is done by solving the issue on your personal fork of the project, and after that making a pull request (PR) with your solution. For more specific details about creating a PR while following the comunity guidelines, check the Contributing page on the Unikraft documentation website.

Example of External Library

Let’s focus for now on an already ported library: lib-libhogweed. Let’s examine its core components. Go to the work/01-tut-porting/libs/libhogweed/ directory and follow the bookmarks marked with USOC_X, where X is the index of the item in the list, from the files specified in the sections below.

Glue Code

In some cases, not all the dependencies of an external library are already present in the Unikraft project, so the solution is to add them manually, as glue code, to the library’s sources.

Another situation when we need glue code is when the ported library comes with test modules, used for testing the library’s functionalities. The goal, in this case, is to wrap all the test modules into one single function. In this way, we can check the library integrity if we want so by just a single function call. Moreover, we can create a test framework which can periodically check all of the ported libraries, useful especially for detecting if a new library will interfere with an already ported one.

Moving back to libhogweed, a practical example of the second case is the run_all_libhogweed_tests(int v) function from libhogweed/testutils_glue.c #674, which calls every selected test module (we will see later how we can make selectable config variables) and returns EXIT_SUCCESS only if it passes over all the tests. For exposing this API, we should also make a header file with all of the test modules, as well as our wrapper function. The header can be found at libhogweed/include/testutils_glue.h.

Config.uk

The Config.uk file stores all the config variables, which will be visible in make menuconfig. These variables can be accessed from Makefile.uk or even from C sources, by including "uk/config.h", using the prefix CONFIG_.

Moving to the source code, libhogweed/Config.uk, we have:

1. The main variable of the library which acts as an identifier for it:

   menuconfig LIBHOGWEED
   	bool "libhogweed - Public-key algorithms"
   	default n
   

2. We can also set another library’s main variable, in this case newlib, which involves including it in the build process:

   select LIBNEWLIBC
   

3. Creating an auxiliary menu, containing all the test cases:

   config TESTSUITE
   		bool "testsuite - tests for libhogweed"
   		default n
   		if TESTSUITE
   			config TEST_X
   				bool "test x functionality"
   				default y
   		endif
   

   Each test case has its own variable in order to allow testing just some functions from the whole suite.

Makefile.uk

The libhogweed/Makefile.uk file is used to:

1. Register the library to Unikraft’s build system:

   $(eval $(call addlib_s,libhogweed,$(CONFIG_LIBHOGWEED)))
   

   As you can see, we are registering the library to Unikraft’s build system only if the main library’s config variable, LIBHOGWEED, is set.

2. Set the URL from where the library will be automatically downloaded at build time:

   LIBHOGWEED_VERSION=3.6
   LIBHOGWEED_URL=https://ftp.gnu.org/gnu/nettle/nettle-$(LIBHOGWEED_VERSION).tar.gz
   

3. Declare helper variables for the most used paths:

   LIBHOGWEED_EXTRACTED = $(LIBHOGWEED_ORIGIN)/nettle-$(LIBHOGWEED_VERSION)
   

   There are some useful default variables, for example:

   • $LIBNAME_ORIGIN: represents the path where the original library is downloaded and extracted during the build process;
   • $LIBNAME_BASE: represents the path of the ported library sources(the path appended to the $LIBS variable).

4. Set the locations where the headers are searched:

   // including the path of the glue header added by us
   LIBHOGWEED_COMMON_INCLUDES-y += -I$(LIBHOGWEED_BASE)/include
   

   You should include the directories with the default library’s headers as well as the directories with the glue headers created by you, if it’s the case.

5. Add compile flags, used, generally, for suppressing some compile warnings and making the build process neater:

   LIBHOGWEED_SUPPRESS_FLAGS += -Wno-unused-parameter \
           -Wno-unused-variable -Wno-unused-value -Wno-unused-function \
           -Wno-missing-field-initializers -Wno-implicit-fallthrough \
           -Wno-sign-compare
   
   LIBHOGWEED_CFLAGS-y   += $(LIBHOGWEED_SUPPRESS_FLAGS) \
           -Wno-pointer-to-int-cast -Wno-int-to-pointer-cast
   LIBHOGWEED_CXXFLAGS-y += $(LIBHOGWEED_SUPPRESS_FLAGS)
   

6. Register the library’s sources:

   LIBHOGWEED_SRCS-y += $(LIBHOGWEED_EXTRACTED)/bignum.c
   

7. Register the library’s tests:

   ifeq ($(CONFIG_RSA_COMPUTE_ROOT_TEST),y)
   LIBHOGWEED_SRCS-y += $(LIBHOGWEED_EXTRACTED)/testsuite/rsa-compute-root-test.c
   LIBHOGWEED_RSA-COMPUTE-ROOT-TEST_FLAGS-y += -Dtest_main=rsa_compute_root_test
   endif
   

   There are situations when the test cases each have their own main() function. In order to wrap all the tests into one single main function, we have to modify their main function name by using preprocessing symbols.

   Note: A good practice is to include a test only if the config variable corresponding to that test is set.

8. This step is very customizable, being like a script executed before starting to compile the unikernel.

   In most cases, and in this case too, the libraries build their own config file through a provided executable, usually named configure:

   $(LIBHOGWEED_EXTRACTED)/config.h: $(LIBHOGWEED_BUILD)/.origin
   	$(call verbose_cmd,CONFIG,libhogweed: $(notdir $@), \
           cd $(LIBHOGWEED_EXTRACTED) && ./configure --enable-mini-gmp \
       )
   LIBHOGWEED_PREPARED_DEPS = $(LIBHOGWEED_EXTRACTED)/config.h
   
   $(LIBHOGWEED_BUILD)/.prepared: $(LIBHOGWEED_PREPARED_DEPS)
   
   UK_PREPARE += $(LIBHOGWEED_BUILD)/.prepared
   

   We can also do things like generating headers using the original building system, modify sources, etc.

Warm-Up

Let’s check the integrity of this library using its test suite through the exposed wrapper function.

For this task, you have to move the library in the $UK_LIBS folder and the work/01-tut-porting/apps/app-libhogweed application in the $UK_APPS folder. You can also keep the current directory structure and clone the Unikraft core repository in the work/0x-task-name directory, basically changing the UK_WORKDIR to work/0x-task-name, as described above. Fill the TODO lines from the application code: add the libhogweed library as a dependency in its Makefile and call the function exposed by the library for running the test suite from main.c. Remember to include the library header file.

Disable some tests, rebuild, and run the checker application again.

Note: The libhogweed library depends on newlib.

Note: Remember to select the test suite from menuconfig. You can also check the library’s README.md for additional information.

Hint: The order of the libhogweed and newlib libraries in the Makefile matters.

Summary

We need a large library pool in order to make the Unikraft project compatible with as many applications as possible.

There are also many ways in which you can contribute to the Unikraft project, and you can find them in the issues section of the main repository.

Practical Work

Support Files

Session support files are available in the repository. If you already cloned the repository, update it and enter the session directory:

$ cd path/to/repository/clone

$ git pull --rebase

$ cd content/en/community/hackathons/sessions/contributing-to-unikraft/

$ ls -F
index.md  work/

If you haven’t cloned the repository yet, clone it and enter the session directory:

$ git clone https://github.com/unikraft/docs

$ cd docs/

$ cd docs/content/en/community/hackathons/sessions/contributing-to-unikraft/

$ ls -F
index.md  work/

00. Prepare

01. Declare Library Identifier

02. Register it to the Build System

03. Set its URL

04. Helper Variables

05. Headers Location

Add the directory which contains the library’s header.

Hint: Inspect $LIBKDTREE_EXTRACTED.

06. Add Sources

Add the library’s C sources.

Hint: Inspect $LIBKDTREE_EXTRACTED.

07. Additional Requirements

Check the original library’s README to see if it needs to be configured first, and add the proper rule if so.

Note: Make sure to also check the library Config.uk file for aditional dependencies.

Hint: As in most ported libraries, a configuration step is required and it consists in simply running a script.

08. Intermediary Check

Until now we have registered the library and its sources, and we should be able to compile an unikernel with it if it doesn’t have any more unresolved dependencies. Move the work/02-task-porting/src/apps/app-kdtree application in the $UK_APPS directory, fill its Makefile, and use it to build an unikernel with our ported library as a dependency!

If needed, provide additional flags in order to suppress the compile warnings generated by this library.

Note: You can leave the application’s main() function empty for now, the resulted unikernel will just print the Unikraft banner.

Hint: Check the tutorial for adding dependencies.

09. Add Test Config Variables

10. Register Test Sources

Moving back to the library’s Makefile.uk, register the tests sources to the build system.

Note: Inspect the functions from the tests.

Hint: Use preprocessing symbols to rename the tests main functions, in order to use them later.

11. Wrapper Glue

Integrate all the test functions into a glue main. Also, update the library’s include/test_suite_glue.h header accordingly.

Note: You can use test_suite_glue.c from the library’s skeleton.

12. Register Glue Code

13. Final Verification

Test the resulted library by calling the test function from the app-kdtree application. If you did everything correctly, the output of the application should look something like this:

SeaBIOS (version 1.13.0-1ubuntu1.1)
Booting from ROM...
Powered by
o.   .o       _ _               __ _
Oo   Oo  ___ (_) | __ __  __ _ ' _) :_
oO   oO ' _ `| | |/ /  _)' _` | |_|  _)
oOo oOO| | | | |   (| | | (_) |  _) :_
 OoOoO ._, ._:_:_,\_._,  .__,_:_, \___)
                   Phoebe 0.10.0~3a997c1
Running test y ....................
inserting 10 random vectors... 0.000 sec
range query returned 0 items in 0.00000 sec
PASS
Running test z ....................
found 5 results:
node at (-3.463, -2.934, 7.719) is 8.108 away and has data=i
node at (-5.316, 2.205, 5.781) is 7.482 away and has data=f
node at (2.210, 3.937, -5.407) is 7.837 away and has data=h
node at (1.810, 8.039, -4.218) is 9.753 away and has data=g
node at (-2.347, -3.641, -7.053) is 9.144 away and has data=e
PASS
Total tests : 2
Total errors: 0

Note: Don’t forget to include the updated library header in the main application file.

14. Give Us Feedback

Further Reading