Studying Macros Was Worth It

51 Days Until I Can Walk

The big win for today has been finally removing the little build.rs file which is responsible for generating lookup tables. While it did its job quite well, I am not a massive fan of the fact that I need to write raw strings out to a .rs file, and then have them included in the compilation of the project. By switching over to using a procedural macro, I can now make use of the quote crate, which gives me just a little more reassurance with regards to the format of the Rust code I am generating, and is generally far more maintainable than the build script option.

Consider, for example, the code below which generates trig tables, first using the build script:

fn main() {
  const SIZE: usize = ANGLE_360 + 1;

  let mut sin: [i32; SIZE] = [0; SIZE];
  let mut cos: [i32; SIZE] = [0; SIZE];
  let mut tan: [i32; SIZE] = [0; SIZE];
  let mut isin: [i32; SIZE] = [0; SIZE];
  let mut icos: [i32; SIZE] = [0; SIZE];
  let mut itan: [i32; SIZE] = [0; SIZE];
  
  for i in 0..=1920 {
      sin[i] = float_to_fix(radian(i).sin());
      cos[i] = float_to_fix(radian(i).cos());
      tan[i] = float_to_fix(radian(i).tan());
      isin[i] = float_to_fix(1.0 / radian(i).sin());
      icos[i] = float_to_fix(1.0 / radian(i).cos());
      itan[i] = float_to_fix(1.0 / radian(i).tan());        
  }

  let mut output = stringify("SIN", &sin, SIZE);
  output.push_str(stringify("COS", &cos, SIZE).as_str());
  output.push_str(stringify("TAN", &tan, SIZE).as_str());
  output.push_str(stringify("ISIN", &isin, SIZE).as_str());
  output.push_str(stringify("ICOS", &icos, SIZE).as_str());
  output.push_str(stringify("ITAN", &itan, SIZE).as_str());

  // generate other tables and push their string representation
  // into output

  let out_dir = env::var("OUT_DIR").unwrap();
  let dest_path = Path::new(&out_dir).join("lookup.rs");
  fs::write(&dest_path, output).unwrap();
}

fn stringify(name: &str, arr: &[i32], size: usize) -> String {
  let mut array_string = String::from("static ");
  array_string.push_str(name);
  array_string.push_str(":[i32; ");
  array_string.push_str(size.to_string().as_str());
  array_string.push_str("] = [\r\n");
  for a in arr {
    // a little bit of formatting is happening as well
    array_string.push_str("\u{20}\u{20}\u{20}\u{20}");
    array_string.push_str(a.to_string().as_str());
    array_string.push_str(",\r\n");
  }
  array_string.push_str("];\r\n");
  array_string
}

This code first generates the trig tables as mutable arrays (so far, so good). However, it then declares a string output and, using the stringify method generates some Rust code that will compile to the same array. This is horrible in my opinion. As we generate lookup tables, we keep appending them to this string, and when we’re done we write the result out to a file with the .rs extension. From there it can be included in a source file in the main project and will be compiled.

Let’s do the same thing using quote

fn declare_trig_tables() -> TokenStream {
  const SIZE: usize    = (consts::ANGLE_360 + 1) as usize;

  let mut sin: [i32; SIZE] = [0; SIZE];
  let mut cos: [i32; SIZE] = [0; SIZE];
  let mut tan: [i32; SIZE] = [0; SIZE];
  let mut isin: [i32; SIZE] = [0; SIZE];
  let mut icos: [i32; SIZE] = [0; SIZE];
  let mut itan: [i32; SIZE] = [0; SIZE];
  
  for i in 0..SIZE {
    sin[i] = (radian!(i).sin()).to_fp();
    cos[i] = (radian!(i).cos()).to_fp();
    tan[i] = (radian!(i).tan()).to_fp();
    isin[i] = (1.0 / radian!(i).sin()).to_fp();
    icos[i] = (1.0 / radian!(i).cos()).to_fp();
    itan[i] = (1.0 / radian!(i).tan()).to_fp() ;
  }

  quote! {
    static SIN: [i32; #SIZE] = [ #(#sin),* ];
    static COS: [i32; #SIZE] = [ #(#cos),* ];
    static TAN: [i32; #SIZE] = [ #(#tan),* ];
    static ISIN: [i32; #SIZE] = [ #(#isin),* ];
    static ICOS: [i32; #SIZE] = [ #(#icos),* ];
    static ITAN: [i32; #SIZE] = [ #(#itan),* ];
  }
}

#[proc_macro]
pub fn insert_lookup_tables(_input: proc_macro::TokenStream) -> proc_macro::TokenStream {
  let trig_tables          = declare_trig_tables();
  
  // generate other tables

  proc_macro::TokenStream::from(quote! {
    #trig_tables
    // other tables
  })
}

First of all, that’s a huge difference in readability! It’s much easier to see what the generate Rust is going to look like by examining the arguments to quote!. This macro directly manipulates the Token Tree generated by the compiler when parsing my code. So there’s no awkward generation of a new Rust file which must be included by the main project. We simply modify the Token Tree resulting from the initial parsing of the main project to now include my lookup tables. Nice!

I do, at this point, very much need to praise the Rust Enhanced plugin for Sublime Text 3. This plugin is constantly checking my code for syntax errors, and in both the build.rs and macro solution to generating lookup tables, the plugin was intelligent enough to recognize that the SIN, COS, TAN etc. arrays to which I was referring would eventually be generated, and so my references to seemingly undeclared variables was not erroneous. That’s excellent work on the part of the developers.

An additional issue with build.rs was that it made use of various constants and functions that were defined in the engine crate. However, because I could not actually import this crate, I ended up duplicating numerous functions (mostly to do with trig and fixed point arithmetic) and constants (projection plane size, angle definitions etc.). Of course, this was still a potential problem when using macros, as the engine needed the macros to generate lookup tables, and the macro needed constants defined in the engine to generate those tables. This circular dependency was solved by introducing a new shared crate, containing constants and functions required both my my macros and the core engine. The resulting project structure looks something like the following.

.
├── Cargo.lock
├── Cargo.toml
├── demo
│   ├── assets
│   │   └── sprites
│   ├── Cargo.toml
│   ├── src
│   │   └── lib.rs
│   └── webapp
│       ├── bootstrap.js
│       ├── demo-level.js
│       ├── favicon.ico
│       ├── index.html
│       ├── index.js
│       ├── package.json
│       ├── package-lock.json
│       ├── vendor
│       │   └── JoyStick
│       └── webpack.config.js
├── engine
│   ├── Cargo.lock
│   ├── Cargo.toml
│   ├── README.md
│   └── src
│       ├── collision.rs
│       ├── lib.rs
│       ├── render.rs
│       ├── scene.rs
│       ├── trig.rs
│       └── utils.rs
├── macros
│   ├── Cargo.toml
│   └── src
│       └── lib.rs
├── shared
│   ├── Cargo.toml
│   └── src
│       ├── consts
│       │   ├── display.rs
│       │   ├── render.rs
│       │   └── trig.rs
│       ├── consts.rs
│       ├── fp.rs
│       └── lib.rs
└── tools
    └── img2tex
        ├── Cargo.toml
        ├── README.md
        └── src
            ├── lib.rs
            └── main.rs

And so, as the great refactor continues, we see the following, rather nice structure emerging:

• shared: The most basic units of the engine. Global constants such as the size of the projection plane, height of the player, etc.
• macros: helper macros used by the engine to generate code at compile time. May make use of constants and functions in shared.
• engine: The engine itself. Defines rendering procedures, game objects, etc.
• demo: A sample game implemented using engine. Should be used to demonstrate the features of the engine.
• tools: A container for miscellaneous tools that are useful for the engine. Currently contains the application used to transform textures from pngs to my internal texture format. Later I expect a desktop application for editing maps will live here too.

Returning to work on my own project with the bit of macro information that I have gleaned was an excellent idea. I have found myself learning little things that I simply wasn’t picking up on when working on the builder. For example, I did not realise that declarative macros live in the root of a crate no matter where they are declared. There are ways around this, but I believe they come with caveats.

One of the advantages that Rust macros have over C macros is that they are significantly smarter than just string substitution. However, they can still bite you in the butt. For example the contents of a macro are execute in the scope of the function that calls them. That means that a macro should fully qualify the namespace of any package contents that it uses. For example, the code below will not work.

// macro.rs

use std::f64::consts::PI;

#[macro_export]
macro_rules! radian {
    ($angle:expr) => {
        $angle as f64 * PI / 180.0
    }
}

// main.rs

fn main() {
  println!("{}", radian!(90));
}

This is because, although macro.rs imports PI, main.rs does not. So when the contents of the macro are substituted into main, PI is not defined. The obvious solution here is to fully qualify PI in the macro:

// macro.rs

#[macro_export]
macro_rules! radian {
    ($angle:expr) => {
        $angle as f64 * std::f64::consts::PI / 180.0
    }
}

// main.rs

fn main() {
  println!("{}", radian!(90));
}

Now this will work.

But let’s say I have my lookup tables and don’t want my users to have direct access to them. So I define a macro to act as a proxy as follows:

// macro.rs

const COS: [i32; ANGLE_360] = [ /* snip */ ];

#[macro_export]
macro_rules! cos {
    ($angle:expr) => {
        COS[$angle as usize]
    }
}

// main.rs

fn main() {
  println!("{}", cos!(90));
}

This cannot work because COS is not visible outside macro. And so the code substitution fails.

This is interesting to me because it highlights in a very small way how macros differ from functions. We get the speed boost of not having to branch to a function, but we need to make some concessions elsewhere.

Tomorrow my plan is to hammer away at finishing the refactor on the renderer. But things are going very well at the moment, so I’m quite optimistic about this.