The Spec's Test Format

The WebAssembly specification is tested with a few hundred files written in an extension of the text format, called WAST (WebAssembly Script), with the extension .wast. A script defines modules and makes assertions about them: that a call returns a certain value, that it traps, that a module fails to validate. Every engine runs these scripts, so the format is supported by many tools, and it is a convenient way to write tests of your own. This chapter covers the format, how to run it, and how a compiler can use it.

#A first script26.1

A script is a sequence of commands. (module ...) defines a module and makes it the current one. assert_return invokes an export of the current module with arguments, and checks the results:

(module
  (func (export "add") (param i32 i32) (result i32)
    (i32.add (local.get 0) (local.get 1)))
  (func (export "div_s") (param i32 i32) (result i32)
    (i32.div_s (local.get 0) (local.get 1))))

(assert_return (invoke "add" (i32.const 1) (i32.const 2)) (i32.const 3))
(assert_return (invoke "add" (i32.const 0x7fffffff) (i32.const 1)) (i32.const 0x80000000))
(assert_return (invoke "div_s" (i32.const -7) (i32.const 2)) (i32.const -3))
(assert_trap (invoke "div_s" (i32.const 1) (i32.const 0)) "integer divide by zero")
(assert_trap (invoke "div_s" (i32.const 0x80000000) (i32.const -1)) "integer overflow")

main.wast: 6 passed, 0 failed

That ran with this book's runner, scripts/wast.py, described in section 26.4. The module counts as one passing command, so there are six.

Arguments and results are written as constant instructions, with the full literal syntax of the text format, so any value can be written exactly: hexadecimal integers, hexadecimal floats, inf, and NaNs with specific payloads.

#The commands26.2

The message strings are what the reference interpreter says. Other runners compare them loosely or not at all, since every engine words its errors differently; what is checked is the kind of failure.

Results can also be patterns. nan:canonical matches a NaN with only the quiet bit set in its payload, which is what arithmetic on non-NaN inputs must produce; nan:arithmetic matches any NaN with the quiet bit set, which is what arithmetic on NaN inputs may produce:

(module
  (func (export "div") (param f32 f32) (result f32)
    (f32.div (local.get 0) (local.get 1)))
  (func (export "add") (param f64 f64) (result f64)
    (f64.add (local.get 0) (local.get 1))))

(assert_return (invoke "div" (f32.const 1) (f32.const 0)) (f32.const inf))
(assert_return (invoke "div" (f32.const 0) (f32.const 0)) (f32.const nan:canonical))
(assert_return (invoke "add" (f64.const nan:0x4) (f64.const 1)) (f64.const nan:arithmetic))
(assert_return (invoke "add" (f64.const 0x1p-1074) (f64.const 0x1p-1074)) (f64.const 0x1p-1073))

main.wast: 5 passed, 0 failed

The last line adds the smallest denormal to itself, which a runner that flushed denormals to zero would get wrong.

Modules can be linked through register, which is how the spec tests imports:

(module $math
  (func (export "square") (param i32) (result i32)
    (i32.mul (local.get 0) (local.get 0))))
(register "math" $math)

(module
  (import "math" "square" (func $square (param i32) (result i32)))
  (func (export "sum_of_squares") (param i32 i32) (result i32)
    (i32.add (call $square (local.get 0)) (call $square (local.get 1)))))

(assert_return (invoke "sum_of_squares" (i32.const 3) (i32.const 4)) (i32.const 25))
(assert_return (invoke $math "square" (i32.const 9)) (i32.const 81))
(assert_unlinkable
  (module (import "math" "cube" (func (param i32) (result i32))))
  "unknown import")

main.wast: 6 passed, 0 failed

And validation is tested with modules that must be rejected:

(assert_invalid
  (module (func (result i32) (i64.const 1)))
  "type mismatch")
(assert_invalid
  (module (func (local.get 0)))
  "unknown local")
(assert_malformed
  (module quote "(func (i32.add_with_typo))")
  "unknown operator")
(assert_malformed
  (module binary "\00asm" "\02\00\00\00")
  "unknown binary version")

main.wast: 4 passed, 0 failed

When an assertion fails, the runner says where and why:

(module
  (func (export "abs") (param i32) (result i32)
    ;; a buggy abs: forgets to negate
    (local.get 0)))
(assert_return (invoke "abs" (i32.const 5)) (i32.const 5))
(assert_return (invoke "abs" (i32.const -5)) (i32.const 5))

main.wast:6: "abs" returned -5, expected (i32.const 5)
main.wast: 2 passed, 1 failed

#Running scripts26.3

Several tools run .wast files:

Browsers' engines run the spec tests too, through harnesses that convert each script into JavaScript. That is the approach this book's runner takes, for the same reason: to test in the engine the code will really run in.

#How the book's runner works26.4

The runner is about four hundred lines in two files, and is worth understanding because it shows how to build test infrastructure on existing tools rather than from scratch.

Splitting. scripts/wast.py parses the script into S-expressions, keeping each one's position in the source. It does not understand the text format at all, only parentheses, strings and comments.

Assembling. Each (module ...) form is cut out of the source as text, and given to an assembler, here the wat crate's by way of Wasmtime's Python package. So every literal, every instruction and every abbreviation is parsed by a real, complete assembler, not by the runner.

Checker modules. The difficult part of a WAST runner is comparing results exactly. The arguments and expected results are text-format constants, which a runner would have to parse; and values like i64s, NaN payloads and v128s cannot cross into JavaScript intact. The runner avoids both problems by turning each assert_return into a small module of its own, in the text format:

(module
  (import "target" "add" (func $f (param i32 i32) (result i32)))
  (global $r0 (export "r0") (mut i32) (i32.const 0))
  (func (export "run") (result i32)
    (call $f (i32.const 1) (i32.const 2))
    (global.set $r0)
    (i32.const 1) (i32.eq (global.get $r0) (i32.const 3)) (i32.and)))

The checker imports the function under test, calls it with the assertion's arguments, and compares each result with the expected value, inside WebAssembly, returning 1 if they all match. The assembler parses the constants, and the comparison is done by WebAssembly instructions: i64.eq for integers, i32.reinterpret_f32 and an integer comparison for exact float bits, bit masks for the NaN patterns, i8x16.eq and i8x16.all_true for vectors. The results are also stored in exported globals, so that when a check fails, the runner can read them and report what the function actually returned.

Executing. The assembled modules, checkers included, are written into a JSON plan, which scripts/wastrun.mjs executes in Node: it instantiates each module, links the checkers to their targets, calls them, and reports.

The lesson generalises. To check something precisely, express the check in the system under test, and let a mature tool parse the inputs.

#Using WAST in a compiler's tests26.5

The spec test suite tests engines, not compilers, but the format is useful to a compiler in three ways.

Testing the runtime library. Parts of a language runtime are often written directly in the text format, or are small enough to test in isolation: memory allocation, string routines, number formatting. A WAST file can test them thoroughly, including traps and edge cases.

Testing generated modules. A compiler can write a test as a WAST file with its own output embedded, using (module binary ...) with the module's bytes, followed by assertions about its exports. Assertions written by hand about a compiled module check the compiler end to end, at the level of WebAssembly values rather than printed output:

import subprocess, wasmtime

# stand-in for your compiler: the "output" here comes from the text format
wasm = wasmtime.wat2wasm('''(module (func (export "fib") (param i32) (result i32)
  (if (result i32) (i32.lt_u (local.get 0) (i32.const 2))
    (then (local.get 0))
    (else (i32.add
      (call 0 (i32.sub (local.get 0) (i32.const 1)))
      (call 0 (i32.sub (local.get 0) (i32.const 2))))))))''')

escaped = "".join(f"\\{b:02x}" for b in wasm)
script = f'''(module binary "{escaped}")
(assert_return (invoke "fib" (i32.const 0)) (i32.const 0))
(assert_return (invoke "fib" (i32.const 1)) (i32.const 1))
(assert_return (invoke "fib" (i32.const 20)) (i32.const 6765))
'''
open("fib.wast", "w").write(script)
print(subprocess.run(["wast.py", "fib.wast"], capture_output=True, text=True).stdout, end="")

fib.wast: 4 passed, 0 failed

Testing a decoder or validator. If your toolchain reads WebAssembly as well as writing it, as a linker, an optimiser or a debugger does, the spec test suite is the most thorough test you will find: its assert_malformed and assert_invalid cases cover every corner of the binary format and the validation rules. The suite lives in the WebAssembly/testsuite repository, with a directory for each proposal.

#What you have seen26.6

WAST scripts define modules and assert facts about them: return values, traps, stack exhaustion, invalid and malformed modules, link errors. Results can be exact, including NaN payloads, or patterns. Many tools run them, and a runner can be built on an existing assembler by turning each assertion into a checker module. A compiler can use WAST to test its runtime library and its generated modules, and the spec test suite to test anything that reads WebAssembly.

The next chapter builds a complete testing strategy for a compiler.