Running C# on my favorite JVM

[lewurm]

During the 2019 Microsoft Hack Week I decided to marry my two favorite VMs: The Mono Runtime and the GraalVM.

Graal is a JIT for the JVM written in Java. This is awesome for itself, but on top of it a whole language framework called Truffle was built. It allows runtime hackers to bring their language to the JVM in a novel way and benefit from high performance. Check it out: https://www.graalvm.org/

The Mono Runtime has a long history on being flexible for all sort of use cases. The latest achievement is that it can run .NET code in the browser via WebAssembly.

Where is the opportunity for marriage here, you may ask?

• The Sulong project implements a Truffle front-end for LLVM bitcode, so you can execute bitcode via Graal.
• The Mono Runtime can emit LLVM bitcode with its AOT compiler.

Obviously someone has to try running Mono generated bitcode on Sulong!

Tl;dr: Sulong doesn't implement pthreads yet, but Mono heavily relies on it. I took a short-cut then and used Mono's interpreter, and got the simplest C# program running with that: return an exit code. And it only needs 8 seconds to do that 😄

Instructions tested on Ubuntu 18.04.

Download GraalVM

The Community Edition (CE) should be enough for our purposes: https://github.com/oracle/graal/releases/tag/vm-19.1.1

$ export PATH=$PATH:/location/to/graalvm-ce-19.1.1/bin
$ lli --version
LLVM (GraalVM CE Native 19.1.1)
$ cat > hello.c
#include <stdio.h>
int main (void) {
    printf ("hello graal\n");
    return 0;
}
$ clang -fembed-bitcode -emit-llvm -c hello.c  # creates hello.bc
$ lli hello.bc
hello graal
$ clang -fembed-bitcode -S -emit-llvm -c hello.c  # creates hello.ll
$ clang -fembed-bitcode -S -c hello.c  # creates hello.s, machine code LLVM would produce.

Nice.

Build a Mono Runtime that runs on Sulong

As you have seen above, Sulong executes LLVM bitcode. In order to get a usable binary out of the Mono build system we can use wllvm, as explained here.

$ sudo apt install clang lldb gettext libtool curl git cmake python build-essential automake autoconf python3-pip 
$ pip3 install wllvm
$ export PATH=$PATH:$HOME/.local/bin
$ git clone [email protected]:lewurm/mono.git -b sulong-mods
$ cd sulong-mods
$ ./autogen.sh LLVM_COMPILER=clang CFLAGS='-O1 -g' CC=wllvm CXX=wllvm++ \
      --with-bitcode=yes --disable-visibility-hidden --disable-boehm \
      --disable-llvm --disable-cxx --prefix=$PWD/b
$ make -j -C mono
$ extract-bc ./mono/mini/mono-sgen

So what was done in this sulong-mods branch? Mainly two things:

1. Stub out all calls to pthreads, as it isn't supported by Sulong. That's quite a bummer, because Mono depends heavily on it. Among other things, it means that we won't have a GC for now and can't use thread local storage. The good news: I was told someone is working on pthread support at this very moment!
2. I sort of lie to autoconf and make it believe to target linux/amd64, and thus I hacked-up a lot of places that are target specific. Specifically, I removed all possible places that emit inline assembly, because Sulong trips over that. In hindsight I should have added another target to Mono, similar to what we did for WebAssembly which has similar constraints.

We can already run the binary generated via wllvm and extract-bc with Sulong:

$ lli ./mono/mini/mono-sgen.bc --version
Mono JIT compiler version 6.4.0 (sulong-mods/dd62f666cd4 Tue Jul 23 05:16:55 PDT 2019)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        Interpreter:   yes
        LLVM:          supported, not enabled.
        Suspend:       hybrid
        GC:            sgen (concurrent by default)

Woot!

Build the BCL

Since the sulong-mods branch is messed up regarding building the base class library (BCL), I recommend to build it in a different checkout:

$ cd ..
$ # 2019-06 is the latest release branch
$ git clone [email protected]:mono/mono.git -b 2019-06 
$ cd 2019-06
$ make -C sdks/builds package-desktop-bcl -j
$ # build result lives in ./sdks/out/desktop-bcl/net_4_x
$ cd ../sulong-mods # let's go back to the other repo checkout

Run stuff!

The lack of pthread is actually a showstopper. That's why I took a short-cut from now on and instead of using Mono's AOT compiler to generate LLVM bitcode for .NET methods as originally intended, I use Mono's interpreter instead.

Let's try the simplest C# program ever:

using System;
public class Hackweek {
   public static int Main (string[] args) {
        return 22;
    }
}

$ export MONO_PATH=../mono-2019-06/sdks/out/desktop-bcl/net_4_x
$ time lli ./mono/mini/mono-sgen.bc --interp exit22.exe; echo $?
lli  --interp   12.50s user 1.06s system 160% cpu 8.428 total
22

It only takes whooping eight seconds! In all fairness, startup performance isn't exactly the strong suite of this whole setup.

Running the same program with --engine.TraceCompilation=true yields some interesting insights on what the dynamic compilation aspect of Sulong/Graal means: https://gist.github.com/lewurm/535ab580c3587aa62e3bc476d44b0728
A Mono Runtime hacker might spot one or other hot function in the runtime 😄

Anything more advanced crashes, I submitted two issues

• [sulong] Error "Cannot convert $val to native pointer" oracle/graal#1518
• [sulong] missing specialization for LLVMGetElementPtrNodeGen oracle/graal#1534

But as I said, without pthread support it isn't worth pushing further.

Debugging hints

lli --inspect allows you to use the Chrome Inspector to debug your bitcode binary. For me this was really only useful in combination with --llvm.lazyParsing=false, so I can properly set breakpoints. Unfortunately that means the whole bitcode binary must contain valid bitcode; that was a bit annoying because I had to remove inline assembly all over the place first.

Another useful flag is --llvm.printStackTraceOnAbort. It does what it says, and I personally think it should be enabled by default.

More information here:

• https://github.com/oracle/graal/blob/master/sulong/docs/DEBUGGING.md
• https://github.com/oracle/graal/blob/443594be722c74ecaba994a0d2a20bc128b78da4/sulong/projects/com.oracle.truffle.llvm.runtime/src/com/oracle/truffle/llvm/runtime/options/SulongEngineOption.java#L99-L100

Conclusion

I'm fascinated by how mature Sulong has become and knowing that an amazing set of people is working on it, I see a bright future for it. Maybe next year's Hack Week I can actually run a real C# programs 🙂

Thanks to Ahmed Shosha and Loïc Sharma to accompany me through the Hack Week! Kudos to the Graal team and specifically thanks to @zapster for answering all my questions about Sulong ❤️