Experimenting with WebAssembly dynamic linking with clang

NOTE: Currently the standard way to compile C or C++ to WebAssembly is to use Emscripten. Additionally, my loader code here is a quick test, and not complete nor necessarily entirely correct.

Background

With LLVM 8.0.0, the WebAssembly back-end is no longer categorized as “experimental”. The LLVM linker, LLD, also supports WebAssembly. So with current releases of LLVM, Clang, and LLD, it’s possible to compile C or C++ to WebAssembly without Emscripten, at the cost of doing things more manually and losing some luxuries like… the standard library. Well, it’s not particularly convenient (without some additional tooling and libraries like Emscripten provides), but it can be done.

WebAssembly 1.0 aims to be fairly minimal. So you might expect that it’s currently limited to static linking. But actually, it’s not; though LLD certainly can statically link WebAssembly. As the WebAssembly documentation explains, the standard provides the basic building blocks to implement dynamic linking, including (to my surprise) run-time linking like dlopen.

A simple example

Linking two WebAssembly modules at load time is actually quite easy, in its basic form:

const fs = require('fs');

async function main() {
    let env = {
        print: console.log
    };

    let wasm1 = fs.readFileSync("module1.wasm");
    let instance1 = (await WebAssembly.instantiate(wasm1, {env: env})).instance;

    Object.assign(env, instance1.exports);
    let wasm2 = fs.readFileSync("module2.wasm");
    let instance2 = (await WebAssembly.instantiate(wasm2, {env: env})).instance;

    instance2.exports.main();
}

main();

I wrote this code, for node.js since it’s a bit more convenient for testing, but browsers offer the same APIs.

We need a module1.c:

int f() {
	return 42;
}

And module2.c:

int f();
void print(int);

int main() {
	print(f());
	return 0;
}

Then we need to compile it:

clang --target=wasm32 -nostdlib -Wl,--no-entry -Wl,--export-all -Wl,--allow-undefined -o module1.wasm module1.c
clang --target=wasm32 -nostdlib -Wl,--no-entry -Wl,--export-all -Wl,--allow-undefined -o module2.wasm module2.c

The meaning of these flags aren’t necessary obvious. The -Wl flags tell Clang to pass an argument to the linker, which is wasm-ld, i.e. LLD targeting WASM.

  • --no-entry tells the linker not to define an entry point. If we don’t pass this flag, LLD defaults to _start, and errors because we haven’t defined that symbol.
  • --export-all exports all symbols, so we can access them from JavaScript (generally, exporting only a specific subset is sensible).
  • --allow-undefined ignores calls to symbols that don’t exist, so we can import them at run time (again, it’s possible to do this only for some symbol names).

And then we can call node on our script, and it works, printing 42.

Some limitations

So this is the basic idea. There are a couple problems:

  1. The module doesn’t contain any information about what modules it depends on. We can manually important each module in the correct order, connecting the exports from one to the imports from the next, but it would be nice to have generic code for this, and include the needed information in the .wasm file itself.
  2. More importantly, currently these modules both create their own, independent memory address space, so pointers passed between them would be incorrect.

Okay, let’s consider that latter point. wasm-ld has a flag that should help. --import-memory build the module to import the linear memory from our JavaScript code, instead of letting the module create one itself. This lets us create a memory address space that our two modules share, which would be important if they passed pointers between each other.

But… it’s still not quite that simple. If we just do that, where do each modules global variables go? Well, probably the same place in memory. For proper dynamic linking, we need to relocate them to the correct address. And even though WASM is a sort of stack machine, Clang requires a stack in memory as well, which is necessary for any variable it takes a pointer to. So both modules need to share a stack somewhere in memory, and a stack pointer.

But as I said at the beginning, WebAssembly has all the basic building blocks needed to implement this. A module can import and export functions and globals, which allows interacting with JavaScript or with other modules. Another simple feature also helps: custom sections allow adding arbitrary data to a .wasm file, so we can include the information needed for proper dynamic linking directly in the binary.

More sophisticated dynamic linking

To provide a standardized design, there is currently a convention for dynamic linking in WebAssembly, though it is not finalized. This is partly implemented by LLD and Clang. Passing --shared to LLD builds as a shared library.

clang --target=wasm32 -nostdlib -Wl,--export-all -Wl,--allow-undefined -Wl,--shared -o module1.wasm module1.c
clang --target=wasm32 -nostdlib -Wl,--export-all -Wl,--allow-undefined -Wl,--shared -o module2.wasm module2.c module1.wasm

And this gives us:

wasm-ld: error: module1.wasm: not a relocatable wasm file
clang-8: error: lld command failed with exit code 1 (use -v to see invocation)

Okay. So, I passed module1.wasm there so it would link against it, similarly to how the -l flag is normally used. (I checked the code of lld, and on wasm currently -l only works with .a static libraries.) But this doesn’t seem to be implented, so it is trying to link module1.wasm as though it’s an object file rather than a shared library.

This works correctly when I build with a git version of LLD, so I guess some needed functionality has been added recently. But this makes it clear that dynamicly linking isn’t really finished in Clang/LLD.

We can look at the resulting binaries using the tools from wabt:

wasm-objdump -x module2.wasm

And here’s a part of the output:

Custom:
 - name: "dylink"
 - mem_size     : 0
 - mem_p2align  : 0
 - table_size   : 0
 - table_p2align: 0
 - needed_dynlibs[1]:
  - module1.wasm

Neat! We now have strings specifying the libraries it links against (currently just module1.wasm). That’s handy. There are no globals, so no relocation there.

We also get a few new imports (which our JavaScript code will have to provide):

 - table[0] elem_type=funcref init=0 max=0 <- env.__indirect_function_table
 - global[0] i32 mutable=1 <- env.__stack_pointer
 - global[1] i32 mutable=0 <- env.__memory_base
 - global[2] i32 mutable=0 <- env.__table_base

LLD also exports a new function called __wasm_call_ctors, which calls an internal function __wasm_apply_relocs. If we don’t override it, it looks like LLD sets this as the start symbol when building a dynamic library. So it will do the relocation for us, as long as we tell it where to using the imports.

Edit: Actually, the entry point is not the start symbol. So we should invoke this function ourselves, which is easy enough.

What is this __indirect_function_table? And for that matter, what is a WebAssembly table? There’s a good Mozilla blog post explaining this. In order to create a pointer to a function in WebAssembly, the function is placed in this table, and the index can be used as a pointer. Thus to share function pointers between the modules, they need to share this table as well as the linear memory. And similar relocation must be applied.

So we can load a dynamic WebAssembly module by doing something like this:

  1. Create a WebAssembly memory and table, and a global for the stack pointer
  2. Allocate a certain amount of the memory for the stack
  3. Compile the module
  4. Parse the dylink custom section
  5. Recursively load the dependencies
  6. Allocate space in the memory and table, according to the dylink section
  7. Instantiate the module, providing the required imports, including the exports from the dependencies

GitHub Repository

The code for this version is a bit long to paste into a blog post, but I’ve created a GitHub repository with it.

Remaining problems

I’ve run into issues trying build C code this way. For instance, when the code contains a static string:

wasm-ld: error: bin.o: relocation R_WASM_MEMORY_ADDR_SLEB cannot be used against symbol .L.str; recompile with -fPIC

This happens even though I compiled with -fPIC, on a build of clang and lld from git. Perhaps I’m doing it wrong, but it looks like a bug in Clang. Which isn’t really entirely unexpected; support for WebAssembly dynamic linking is incomplete currently.

Runtime dynamic linking (dlopen)

I haven’t implemented dlopen, but the idea is simple enough. Instantiate the new module, then the functions can be passed to an already running module by placing them in a table and providing the index in the table to the existing module.

Conclusion

You probably don’t want to use wasm-ld for dynamic linking in it’s current state, though it should be nice once the support in Clang and LLD is matured. And really, most users probably want to rely on a project like Emscripten that provides conveniences like the C standard library (which is sort of important).

But it’s quite neat how WebAssembly makes things like this possible using a fairly minimal and elegant design: high level enough that the mechanisms are easy to use, but low level enough to not commit to too specific of a design.

And of course if you want to mess around with WebAssembly at a low level, for practical reasons or otherwise, you certainly can!