Look Ma, No wasm-pack

67 Days Until I Can Walk

For this essay, I want to dive a little bit into what wasm-pack is doing behind the scenes. In the words of the developers themselves:

[wasm-pack] seeks to be a one-stop shop for building and working with rust-generated WebAssembly that you would like to interop with JavaScript, in the browser or with Node.js. wasm-pack helps you build rust-generated WebAssembly packages that you could publish to the npm registry, or otherwise use alongside any javascript (sic.) packages in workflows that you already use, such as webpack.

I have been using wasm-pack for two weeks now, and it has made it trivially simple for me to go from Rust code to a usable web application without needing to worry too much about exactly how that transformation works. This is all well-and-good if you understand the processes that wasm-pack is abstracting away from you, but treating it like a black-box with no understanding of what is happening inside strikes me as willful ignorance which really should be addressed.

Perhaps the best way to understand exactly what wasm-pack is doing would be to attempt to build a WASM project without using wasm-pack, but whilst carrying out the same set of instructions. In order to do this, I cloned the wasm-pack repository and read through the code for the build command. Occasionally I dropped in a println! statement and recompiled wasm-pack so I could inspect the value of certain variables.

At 3453 lines of Rust code (according to cloc), wasm-pack is not an enormous project, and there is a lot to be learned by inspecting the repo. The crate was started by Ashley Williams (formerly of the Rust Core team and founder of the Rust Foundation), and is currently maintained by Jesper Håkansson, aka drager. These are two heavy-hitters in the Rust community. Following their coding style is an incredibly interesting exercise and taught me a lot about idiomatic Rust. Additionally, I discovered a number of useful crates while working through their code. These I have listed at the bottom of this article.

As always, I am learning about these things as I write, so do not take this article as an authoritative source. Read with a critical eye.

The Big Picture

If we inspect the wasm_pack::command::build module of the project, we can find the code which executes the build command in the Build::run method. Inspecting this, we can see that by default the wasm-pack build command will perform the following steps in order:

1. Check Rust Compiler Version
2. Validate The Current Crate
3. Check For A Wasm Target
4. Build The WASM File
5. Create Output Project Directory
6. Install wasm_bindgen
7. Run wasm-bindgen
8. Run wasm-opt
9. Create A package.json File
10. Copy project README to the output directory
11. Copy project LICENSE to the output directory

Let’s take these in turn.

Check Rust Compiler Version

Nothing fancy happening here. If you want to check yourself what version of the Rust compiler you have, execute

rustc -V

wasm-pack requires that you at least have Rust version 1.30.0 installed, so if you’re running an older version it’s definitely time to update.

rustup update

Validate The Current Crate

During initialization wasm-pack extracts and parses information about your package from Cargo.toml. At this stage of the build your project configuration is checked to ensure that your crate type is cdylib. If you do not have the correct crate type, wasm-pack will recommend the following course of action:

crate-type must be cdylib to compile to wasm32-unknown-unknown. Add the following to your Cargo.toml file:
 
[lib]
crate-type = ["cdylib", "rlib"]"

Setting your crate-type to cdylib tells cargo that you want your project to be built as a dynamic linked library (*.so on Linux, *.dylib on macOS, and *.dll on Windows). According to the linkage chapter of The Rust Reference, this is specifically for when you want your dynamic library to be loaded from other languages. So this makes sense for building our WASM file to be loaded by JavaScript.

If your Cargo.toml file is missing the above crate type, add it now.

Check For A Wasm Target

Next wasm-pack will check to ensure that you have the wasm32-unknown-unknown target installed. Targets allow you to cross-compile your Rust code for other platforms. By default you only have a target for your current platform installed, but using rustup you can install targets so that you can compile for other architectures e.g. ARM. This is discussed in the cross-compilation section of The rustup Book.

Why is the target called wasm32-unknown-unknown? The answer, at least according to this GitHub issue seems to be that the first unknown refers to the system you are compiling on and the second unknown refers to the target system. To quote the answer by chinedufn

So you an think of unknown-unknown as

“Compile on almost any machine, run on almost any machine”

If you look at other targets “unknown” is commonly used so this is in line with the other targets.

So the next thing you need to do is check if you have the wasm32-unknown-unknown target installed, and if not you need to set it up. You can do this by getting rustup to list all available targets, and then grepping for those that are installed:

rustup target list | grep installed

If wasm32-unknown-unknown is not installed, add it with the following command:

rustup target add wasm32-unknown-unknown

Build The WASM File

With these initial checks complete, wasm-pack calls cargo and parameterizes it to compile your project to WebAssembly. The command generated will look something like the following:

cargo build --lib --target wasm32-unknown-unknown

Congratulations! You’ve just compiled your Rust code into a WASM file!

cargo will store the result of your build in the target directory.

Create Output Project Directory

Fairly self explanatory. wasm-pack creates a directory where it will store the final compiled WASM and associated JavaScript files. By default this directory is called pkg. If a pkg directory from a previous run exists, wasm-pack will delete the old package.json file from this folder.

wasm-pack will also add a .gitignore file to the pkg directory. This tells git to ignore all contents of the pkg directory.

If you want to emulate this behaviour, the following will do the trick

rm -f pkg/package.json
mkdir -p pkg
echo "*" > pkg/.gitignore

Install wasm-bindgen

wasm-bindgen is another tool in the Rust and WebAssembly ecosystem. Its job is to creating bindings between JavaScript and Rust code. If you have written a WebAssembly project in Rust before, then you will likely recall the #[wasm_bindgen] annotation applied to functions and structs that you wish to expose to your JavaScript code. wasm-pack uses a command line utility called wasm-bindgen-cli during the next phase of the build. The documentation for wasm-bindgen simply states that:

The wasm-bindgen command line tool has a number of options available to it to tweak the JavaScript that is generated.

wasm-pack will first check to see if a global installation of wasm-bindgen-cli with the correct version is already installed. The version number is determined by the version for wasm-bindgen given in your Cargo.toml file—you must, of course, have wasm-bindgen listed as a dependency in order to apply the #[wasm_bindgen] annotation. If such a global installation is found, then wasm-pack will use that version in the next step.

If wasm-bindgen-cli is not installed, then wasm-pack will use the binary-install crate to download and install the correct version. This download will not be installed globally, but rather is downloaded to a cache folder ($HOME/.cache/.wasm-pack on Linux).

Assuming a suitable version of wasm-bindgen-cli is found and/or installed, wasm-pack will proceed to the next step.

For our purposes, we will install wasm-bindgen-cli globally.

cargo install wasm-bindgen-cli

Run wasm-bindgen

With wasm-bindgen-cli installed, wasm-pack will use the utility to prepare your Rust project for deployment to the web. Appropriate JavaScript files which can be loaded in a browser or imported into a Node module will be generated. The equivalent command to that executed by wasm-pack is

wasm-bindgen target/wasm32-unknown-unknown/release/<your-project>.wasm --out-dir pkg --typescript --target bundler

By default the target is bundler and typescript is enabled, but wasm-pack explicitly sets them, so we will too. Detailed information about available flags can be found in the wasm-bindgen commandline and deployment documentation.

Run wasm-opt

The final stage of the build itself is to optimise the resulting wasm file using wasm-opt. As before, wasm-pack first checks to see if wasm-opt is installed globally, and if not it will download an appropriate version and install it to a cache. We’ll install the binary globally.

cargo install wasm-opt

Now we can optimise our wasm file. For every generated wasm file, wasm-pack executes the following instruction:

wasm-opt pkg/<your-project>.wasm -o pkg/<your-project>.wasm-opt.wasm -O

The -O flag in this context tells wasm-opt to use the default optimisation level which, at the time of writing is -Os—execute default optimization passes, focusing on code size. The new, optimised wasm file will then be copied over the old one.

mv pkg/<your-project>.wasm-opt.wasm pkg/<your-project>.wasm

Create A package.json File

The actual build of your crate is now done, but wasm-pack will proceed to generate a package.json file for your project. Exactly how this happens depends on what target was specified for wasm-bindgen. By default this is bundler.

wasm-pack will attempt to populate the following fields, and will obtain their values as described:

• name: The output name of the project (probably the Cargo.toml name attribute), but if your project is scoped then it will be in the format “@{scope}/{name}”
• collaborators: Obtained from Cargo.toml authors,
• description: Obtained from Cargo.toml description,
• version: Obtained from Cargo.toml version,
• license: Obtained from Cargo.toml license(),
• repository: Obtained from Cargo.toml repository,
• files: Obtained by scanning the contents of the pkg directory for *_bg.wasm, *_bg.js, *.js files,
• module: Main JavaScript file of the build. Of the format .js where `name` is the field described above,
• homepage: Obtained from Cargo.toml homepage,
• types: Obtained from pkg directory—look for a file ending in *.d.ts,
• side_effects: Hardcoded to a list containing “./” + module from above, and “./snippets/*”,
• keywords: Obtained from Cargo.toml keywords,
• dependencies: Obtained from an existing package.json file if one is found,

A sample output, should you wish to replicated it, would look something like the following

{
  "name": "<your-project-name>",
  "version": "0.1.0",
  "files": [
    "<your-project-name>_bg.wasm",
    "<your-project-name>.js",
    "<your-project-name>_bg.js",
    "<your-project-name>.d.ts"
  ],
  "module": "<your-project-name>.js",
  "types": "<your-project-name>.d.ts",
  "sideEffects": [
    "./<your-project-name>.js",
    "./snippets/*"
  ]
}

Copy Project README To The Output Directory

wasm-pack will copy your crate’s README as specified in your Crate.toml file to the pkg directory. If no README is found, then an error message will be printed.

cp <your-readme> pkg

Copy Project LICENSE To The Output Directory

wasm-pack will copy your crate’s license file to the pkg directory. wasm-pack will search your root crate directory for files matching the pattern LICENSE-*. If no license is found, then an error message will be printed.

cp <your-license> pkg

Conclusion

That concludes the steps that wasm-pack executes when running the build subcommand. As can be seen, there are several tools running under the hood that are easy to miss if you treat wasm-pack as a black-box. It would be interesting to dive more into wasm-bindgen, wasm-opt, or even some of the other wasm-pack subcommands at some point in the future.

Useful Crates

One of the benefits of reading through someone else’s source code is that you can discover new libraries and programming techniques. Reading through wasm-pack, I discovered the following crates which I would like to remember and possibly use at some point in the future.

env_logger: This crate provides command line logging functionality in Rust. The logger can be configured using environment variables and command line arguments. Coming from something of a Java background myself, this crate seems like a good analogue to the log4j package.

clap: Used for parsing command line arguments. I have actually seen clap before in one of Tim McNamara’s tutorial videos. This is a very handy crate for building interfaces to CLI tools.

human-panic: A crate which produces a call-to-action message if the application crashes. A message with details of how to report bugs, and a file containing stacktrace information is generated to encourage users to send helpful information to developers which will enable them to fix bugs.

console: Used for styling output to the command line (colours, bold, italic, etc.)

cargo_metadata: Parses and gives you access to the output of cargo metadata for an input Cargo.toml file. This allows you to inspect the dependencies and configuration for a crate.

binary-install: Provides utilities for downloading and installing crates from within Rust code.