In the previous sessions, we have explored porting a simple application to Unikraft. That required heavily changing the source code, which can lead to broken behavior. Today, we will learn how to properly port a more complex application, using multiple source file and build steps.
We will use iperf3 as an example, a network benchmarking tool.
The lifecycle of the construction of a Unikraft unikernel includes several distinct steps:
The steps in the lifecycle above are discussed in this tutorial in greater depth.
Particularly, we cover fetching, preparing and compiling (building) external code which is to be used as a Unikraft unikernel application (or library for that matter).
For the sake of simplicity, this tutorial will only be targeting applications which are C/C++-based. Unikraft supports other compile-time languages, such as Golang, Rust and WASM. However, the scope of this tutorial only follows an example with a C/C++-based program. Many of the principles in this tutorial, however, can be applied in the same way for said languages, with a bit of context-specific work. Namely, this may include additional build rules for target files, using specific compilers and linkers, etc.
It is worth noting that we are only targeting compile-time applications in this tutorial. Applications written a runtime language, such as Python or Lua, require an interpreter which must be brought to Unikraft first. There are already lots of these high-level languages supported by Unikraft.(e.g., python) If you wish to run an application written in such a language, please check out the list of available applications. However, if the language you wish to run is interpreted and not yet available on Unikraft, porting the interpreter would be in the scope of this tutorial, as the steps here would cover the ones needed to bring the interpreter, which is a program after all, as a Unikraft unikernel application.
For the remainder of this tutorial, we will be targeting the network utility program iperf3 as our application example we wish to bring to Unikraft.
iperf3 is a benchmarking tool, and is used to determine the bandwidth between a client and server.
It makes for an excellent application to be run as a Unikernel because:
server-type application, receiving and processing requests for clientsBringing an application to Unikraft will involve understanding some of the way in which the application works, especially how it is built. Usually during the porting process we also end up diving through the source code, and in the worst-case scenario, have to make a change to it. More on this is covered later in this tutorial.
We start by simply trying to follow the steps to compile the application from source, for our usual Linux setup, with no connection to Unikraft for now.
The README for the iperf3 program has relatively simple build instructions, and uses GNU Make which is a first good sign.
Unikraft uses GNU Make to handle its internal builds and so when we see an application using Make, it makes porting a little easier.
For non-Make type build systems, such as CMake or Bazel, it is still possible to bring this application to Unikraft, but the flags, files, and compile-time options, etc.
will have to be considered with more care as they do not necessarily align in the same ways.
It is still possible to bring an application using an alternative build system, but you must closely follow how the program is built in order to bring it to Unikraft.
Let's walk through the build process of iperf3 from its README:
First we obtain the source code of the application:
git clone https://github.com/esnet/iperf.git
Then, we are asked to configure and build the application:
cd ./iperf./configure;make
This will generate the iperf3 executable, located in src/iperf3.
If this has worked for you, your terminal will be greeted with several pieces of useful information:
The first thing we did was run ./configure: an auto-generated utility program part of the automake build system.
Essentially, it checks the compatibility of your system and the program in question.
If everything went well, it will tell us information about what it checked and what was available.
Usually this ./configure-type program will raise any issues when it finds something missing.
One of the things it is checking is whether you have relevant shared libraries (e.g. .so files) installed on your system which are necessary for the application to run.
The application will be dynamically linked to these shared libraries and they will be referenced at runtime in a traditional Linux user space manner.
If something is missing, usually you must use your Linux-distro's package manager to install this dependency, such as via apt-get.
The ./configure program also comes with a useful --help page where we can learn about which features we would like to turn on and off before the build.
It's useful to study this page and see what is available, as these can later become build options for the application when it is brought to the Unikraft ecosystem.
If, however, there are library dependencies for the target application which do not exist within the Unikraft ecosystem, then these library dependencies will need to be ported first before continuing.
The remainder of this tutorial also applies to porting libraries to Unikraft.
When we next run make in the sequence above, we can see the intermediate object files which are compiled during the compilation process before iperf3 is finally linked together to form a final Linux user space binary application.
It can be useful to note these files down, as we will be compiling these files with respect to Unikraft's build system.
You have now built iperf3 for Linux user space and we have walked through the build process for the application itself.
In the next section, we prepare ourselves to bring this application to Unikraft.
Applications which are brought to Unikraft are actually libraries.
Everything in Unikraft is libracized, so it is no surprise to find out that even applications are a form of library.
They are a single component which interact with other components, have their own options and build files and interact in the same ways in which other libraries interact with each other.
The main difference between actual libraries and applications, is that we later invoke the application's main method.
The different ways to do this are covered later in this tutorial.
To get started, we must create a new library for our application.
The premise here is that we are going to wrap or decorate the source code of iperf3 with the lingua franca of Unikraft's build system.
That is, when we eventually build the application, the Unikraft build system will understand where to get the source code files from, which ones to compile and how, with respect to the rest of Unikraft's internals and other dependencies.
We will start from the nginx application, since it has the same requirements as iperf3, as in a libc and a networking stack.
First, create an empty directory under workdir/libs/, called iperf3:
$ mkdir wordir/libs/iperf3/
Next, we need to create the 2 most relevant files for the Unikraft build system, Config.uk and Makefile.uk:
$ touch workdir/libs/iperf3/Config.uk$ touch workdir/libs/iperf3/Makefile.uk
The Makefile.uk should have minimal details about the location of the iperf3 archive online:
################################################################################# Library registration################################################################################$(eval $(call addlib_s,libiperf3,$(CONFIG_LIBIPERF3)))################################################################################# Original sources################################################################################LIBIPERF3_VERSION=3.19LIBIPERF3_BASENAME=iperf-$(LIBIPERF3_VERSION)LIBIPERF3_URL=https://github.com/esnet/iperf/archive/refs/tags/$(LIBIPERF3_VERSION).tar.gz$(eval $(call fetch,libiperf3,$(LIBIPERF3_URL)))
The Config.uk file will contain one option that will allow us to later select the library from the menuconfig screen:
config LIBIPERF3bool "lib iperf 3.14"default y
Now, we can modify the nginx Makefile and replace $(LIBS_BASE)/nginx with $(LIBS_BASE)/iperf3, which will load the iperf3 Makefile.uk file.
After that, if we run make menuconfig, we should have a iperf3 option under Library Configuration -->.
If we select that, we can run make fetch to download the source code of iperf for our application:
$ make menuconfig# Select Library Configuration --> lib iperf 3.14 and save$ make fetchmake[1]: Entering directory ...LN MakefileWGET libiperf3: https://github.com/esnet/iperf/archive/refs/tags/3.14.tar.gz.../app-iperf/build/libiperf3/3.14.tar.gz [ <=> ] 635,38K 2,66MB/s in 0,2sUNTAR libiperf3: 3.14.tar.gzmake[1]: Leaving directory ...$ tree -L 2 workdir/build/libiperf3/workdir/build/libiperf3/|-- 3.14.tar.gz|-- origin| `-- iperf-3.14`-- uk_clean_list2 directories, 2 files
The next thing we need to do is provide source files that need to be built for libiperf3 to work.
Now we have everything set up. We can start an iterative process of building the target unikernel with the application. This process is usually very iterative because it requires building the unikernel step-by-step, including new files to the build, making adjustments, and re-building, etc.
The first thing we must do before we start is to check that fetching the remote code for iperf3 worked.
The directory with the extracted contents should be located at:
$ ls -lsh workdir/build/libiperf3/origin/iperf-3.14/total 1,0M372K -rw-r--r-- 1 stefan stefan 367K iul 8 00:47 aclocal.m44,0K -rwxr-xr-x 1 stefan stefan 1,5K iul 8 00:47 bootstrap.sh4,0K drwxr-xr-x 2 stefan stefan 4,0K iul 8 00:47 config504K -rwxr-xr-x 1 stefan stefan 499K iul 8 00:47 configure12K -rw-r--r-- 1 stefan stefan 11K iul 8 00:47 configure.ac4,0K drwxr-xr-x 2 stefan stefan 4,0K iul 8 00:47 contrib4,0K drwxr-xr-x 3 stefan stefan 4,0K iul 8 00:47 docs4,0K drwxr-xr-x 2 stefan stefan 4,0K iul 8 00:47 examples12K -rw-r--r-- 1 stefan stefan 9,3K iul 8 00:47 INSTALL4,0K -rw-r--r-- 1 stefan stefan 1,5K iul 8 00:47 iperf3.spec.in12K -rw-r--r-- 1 stefan stefan 12K iul 8 00:47 LICENSE4,0K -rw-r--r-- 1 stefan stefan 23 iul 8 00:47 Makefile.am28K -rw-r--r-- 1 stefan stefan 26K iul 8 00:47 Makefile.in4,0K -rwxr-xr-x 1 stefan stefan 1,2K iul 8 00:47 make_release8,0K -rw-r--r-- 1 stefan stefan 6,4K iul 8 00:47 README.md36K -rw-r--r-- 1 stefan stefan 36K iul 8 00:47 RELNOTES.md4,0K drwxr-xr-x 2 stefan stefan 4,0K iul 8 00:47 src4,0K -rwxr-xr-x 1 stefan stefan 2,0K iul 8 00:47 test_commands.sh
If this has not worked, you must fiddle with the preamble at the top of the library's Makefile.uk to ensure that correct paths are being set.
Remove the build/ directory and try make fetch again.
Now that we can fetch the remote sources, cd into this directory and perform the ./configure step as above.
This will do two things for us.
The first is that it will generate (and this is very common for C-based programs) a config.h file:
$ cd workdir/build/libiperf3/origin/iperf-3.14/$ ./configure[...]config.status: creating src/version.hconfig.status: creating examples/Makefileconfig.status: creating iperf3.specconfig.status: creating src/iperf_config.hconfig.status: executing depfiles commandsconfig.status: executing libtool commands$ ls -sl src/iperf_config.h8 -rw-rw-r-- 1 stefan stefan 4246 iul 10 19:51 src/iperf_config.h
This file is a list of macro flags which are used to include or exclude lines of code by the preprocessor. If the program has one of these, we need it.
Let's copy this file into our Unikraft port of the application.
Make an include/ directory in the library's repository and copy the file:
mkdir ~/workdir/app-iperf/workdir/libs/iperf3/includecp workdir/build/libiperf3/origin/iperf-3.10.1/src/iperf_config.h ~/workdir/app-iperf/.unikraft/libs/iperf3/include
Let's indicate in the Makefile.uk of the Unikraft library for iperf3 that this directory exists, and should be used as a location to look for header files:
Add this line in the workdir/libs/iperf3/Makefile.uk file:
LIBIPERF3_CINCLUDES-y += -I$(LIBIPERF3_BASE)/include
We'll come back to iperf_config.h;
likely it needs edits from us to turn features on or off depending on availability or applicability based on the unikernel-context.
Next, let's run make with a special flag.
The make might give an error at the end, but it's fine, we can ignore it.
cd workdir/build/libiperf3/origin/iperf-3.14/make -n
This flag, -n, has just shown us what make will run, the full commands for gcc including flags.
What's interesting here is any line which starts with:
echo " CC "
These are lines which invoke gcc.
We can gather a few pieces of information here, namely the flags and list of files we need to make iperf3 a reality.
Let's start by setting global flags for iperf3.
The rule of thumb here is that we copy the flags which are used in all invocations of gcc and place them within the Makefile.uk.
We should ignore flags to do with optimization, PIE, shared libraries and standard libraries as Unikraft has global build options for these.
Flags which are usually interesting are to do with suppressing warnings (e.g. things that start with -W) and are application-specific.
There doesn't seem to be anything immediately obvious for iperf3.
However, in a later step, we'll find out that we can set some flags.
If you do have flags which are immediately obvious, you set them like so in the library port's Makefile.uk, for example:
LIBIPERF3_CFLAGS-y += -Wno-unused-parameter
We have a full list of files for iperf3 from the previous step.
We can add them as known source files like so to the Unikraft port of iperf3's Makefile.uk:
LIBIPERF3_SRCS-y += $(LIBIPERF3_SRC)/main.cLIBIPERF3_SRCS-y += $(LIBIPERF3_SRC)/cjson.cLIBIPERF3_SRCS-y += $(LIBIPERF3_SRC)/iperf_api.cLIBIPERF3_SRCS-y += $(LIBIPERF3_SRC)/iperf_error.c...
Note: The path in the variable LIBIPERF3_SRC may need to be adjusted from the boilerplate code to match the layout of the application you are porting.
Note: Some of the source files that are compiled might be test files, we should not add them to the Unikraft build system.
Tip: It's best to add these files iteratively (i.e. one by one) and attempt the compilation process (next step) in between adding all files. This will show you errors about what's missing, and you can accurately determine which files are truly necessary for the build. In addition to this, we can also find intermittent errors which will be the result of incompatibilities between Unikraft and the application in question (covered in the next section on making patches).
Now that we have added all the source files, let's try and build the application! This step, again, usually occurs iteratively along with the previous step of adding a new file one by one. Because the application has been configured and we have fetched the contents, we can simply try running the build in the Unikraft application directory:
cd ~/workdir/app-iperf3make
(Optional) This step occurs less frequently, but is still useful to discuss in the context of porting an application to Unikraft.
Remember in the Unikraft build lifecycle that there is a step which occurs between fetching the remote original code and compiling it.
This step (3), known as prepare, is used to make modifications to the origin code before it is compiled.
This may be useful for applications which have complex build systems or auxiliary files which need to be created or modified before they are built.
Examples for preparing include:
Preparation is done by adding Make targets to the UK_PREPARE variable:
UK_PREPARE += mytarget
Checking whether the library has been prepared or adding a target which requires preparation before it can be executed is as simple as checking whether the following target exists:
$(LIBIPERF3_BUILD)/.patched
The prepare step is called naturally because of this target.
However, it can be called separately from make via:
make prepare
The steps outlined above helped us begin the process of porting a simple application to Unikraft.
It covers the major steps involved in the process of porting from first principles, including addressing all the steps in the construction lifecycle of Unikraft unikernels.
There are occasional caveats to this process, however.
This is to do with the context of the unikernel model, that is single-purpose OSes with a single address space, acting in a single process without context switches or costly syscalls.
Applications developed for Linux user space make a number of assumptions about its runtime, for example:
Patching the application occasionally must occur to address incompatibilities with the context of a Linux user space application and that of the unikernel model. It can also be used to introduce new features to the application, although this is rare (although, here is an example).
Identifying a change to the application which requires a patch is sometimes quite subtle.
The process usually occurs during steps 5 and 6 of providing build files of the application or library in question.
During this process, we are expected to see compile-time and link-time errors from gcc as we add new files to the build and make fixes.
The iperf3 application port to Unikraft has four patches in order to make it work.
Let's discuss them and what they mean.
The next section discusses how to create one of these patches.
The first patch comes from an error which is thrown when compiling the iperf_api.c source file.
This file is 3rd to be compiled from the list of complete source files.
In this file, we are receiving a duplicate import of <netinent/tcp.h>, simply removing this import fixes it, so the patch addresses this issue.
The second patch comes as a result of missing functionality from LwIP.
The issue was discovered once the application was fully ported and was able to boot and run.
When the initialization sequence was on-going between the client and server of iperf3, it would crash during this sequence because LwIP does not support setting this option.
A patch was created simply to remove setting this option.
(Note: this may not be the most sensible approach)
The third patch arises from an assumption about the host environment and the difference between Linux user space and a unikernel.
With a traditional host OS, we have a filesystem populated with known paths, for example /tmp.
iperf3 assumed this path exists, however, in the case of where no filesystem is provided to the unikernel during boot, which should be possible in some cases, the iperf3 application would crash since /tmp does not exist beforehand.
The patch solves this by setting the temporary (ramfs) path to /.
An alternative solution is to make this path at boot.
The above patches represent example use cases where patches may be necessary to fix the application when bringing it to Unikraft. The possibilities presented in this tutorial are non-exhaustive, so take care. The next section discusses in detail how to create a patch for the target application or library.
When a change is identified and is to be provided as a patch to the application or library during the compilation, it can be done using the procedure identified in this section. Note that providing patches are an unfortunate workaround to the inherent differences between Linux user space applications and libraries and unikernels.
Note: When patches are created, they are also version-specific.
As such, if you update the library or application's code (i.e. by updating, for example, the version number of LIBIPER3_VERSION), patches may no longer be apply-able and will then need to be updated accordingly.
To make a patch:
First, ensure that the remote origin code has been downloaded to the application's build/ folder:
cd ~/workspace/apps/iperf3make fetch
Once the source files have been downloaded, turn it into a Git repository and save everything to an initial commit, in the case of iperf3:
cd workdir/build/libiperf3/origin/iperf-3.10.1git initgit add .git commit -m "Initial commit"
This will allow us to make changes to the source files and save those differences.
After making changes, create a Git commit, where you briefly describe the change you made and why. This can be done through a number of successive steps, for example, as a result of having to make several changes to the application.
After your changes have been saved to the git log, export them as patches.
For example, if you have made one (1) patch only, export it like so:
git format-patch HEAD~1
This will save a new .patch file in the current directory; which should be the origin source files of iperf3.
The next step is to create a patches/ folder within the Unikraft port of the library and to move the new .patch file into this folder:
mkdir ~/workspace/libs/iperf3/patchesmv ~/workspace/apps/iperf3/build/libiperf3/origin/iperf-3.10.1/*.patch ~/workspace/libs/iperf3/patches
To register patches against Unikraft's build system such that they are applied before the compilation of all source files, simply indicate it in the library's Makefile.uk:
# Add or edit ~/workspace/libs/iperf3/Makefile.ukLIBIPERF3_PATCHDIR = $(LIBIPERF3_BASE)/patches
Feel free to ask questions, report issues, and meet new people.