Shimlang has been written in Python, Zig, Rust, Zig, and Rust again. Each step has improved the implementation has added crucial functionality.
It’s now time for another rewrite. Primarily, this rewrite will replace the custom bytecode interpreter with a WASM compiler.
The most recent Shimlang Rust implementation does not handle intermediate
heap-allocated values well. Recently I was creating an sdf font using the existing Rust implementation of Shimlang. For convenience I was doing this
all on a single frame at startup. Because I was using tuples (which need to
allocate) this ended up busting the 2^27 (134MB) available. I had 64
128x128 images with a 2-tuple (16 bytes) for each pixel. There were at
least 8 sdf operations per pixel. Thats 2^(6+7+7+4+3) = 2^27. Even with
no other operations, this hits the limit.
The implementation doesn’t support running GC mid-way through executing a frame. To do that, it would need to track every temporary value. Despite the limitations we can see with this approach, avoiding the runtime complexity of tracking these values is more important.
Another approach is to do some form of “escape analysis”. This would mean tracking which allocations can be freed statically. The bytecode compiler could track per-function which non-value arguments are assigned to globals or other input arguments or captured in returned closures. From there it would be able to free heap-allocated data as part of the generated bytecode.
That approach would give much more headroom to the sorts of value-heavy code that is likely to bust the memory limit.
The other approach is to avoid heap allocations. In particular, tuples are only heap allocated since the runtime can only handle 8-byte values outside of the heap. Changing that approach requires a complete rewrite of the bytecode compiler and runtime.
So we’re left with some options:
Each of those requires a fair amount of work. But there’s one last thing, we don’t even want to use the current runtime in production. Shimlang needs to support WASM to have any hope of reasonable web performance.
So we need to change the compiler and we need to change the runtime. So much needs to change that it would be most-effective to start from scratch. As with previous rewrites, the corpus of tests will be carried forward.
Creating a language that’s meant for use in a game engine essentially requires using a systems programming language. The actual game engine part of it needs to be fast and we need to integrate with existing C/C++ libraries for physics, graphics, audio, and input.
To me, it comes down to these languages:
To me, Zig/Odin are more attractive than C. Despite never writing Odin, I still find it more readable than C and it comes with many features that seem tailor-made for game development.
I find that I don’t experience the joy of coding when I write in Zig.
I find that I don’t enjoy using the standard library. Part of that is the
allocator interface but another part of it is the hard-to-follow
documentation. Take a look at the documentation for ArrayList.
How do you create one? How do you push to one? How do you access a pushed
value? I think I want an (std.array_list.Aligned)[https://ziglang.org/documentation/master/std/#std.array_list.Aligned]?
Why does .append take an allocator rather than the allocator being stored
in the list?
The other main issue I have with Zig is the build system. It seems to change with every release. It also feels very bad to write what should be a script in a low level language. I get the academic appeal of having the build scripting being in the same language, but it ends up being a pain.
I can’t write C++, unless I use a very small subset which eliminates a lot of the ergonomic benefits you would hope to get.
Rust has worked fine, but fights you when you want to operate with raw memory. So much of Rust is built around thread safety and making it hard to write incorrect code. In this use-case that safety is getting in the way more often then not. I also want to use globals. It’s a pain to have to use locks when I know there’s only a single thread accessing some data.
D seems fine. I have a coworker I greatly respect who is a D enjoyer. It just doesn’t have the mindshare that I would have expected if it were good for this niche.
Beef/Lobster are their own runtime. Interesting, but I want to make the runtime.
The creator of Jai doesn’t like people writing their own programming language in Jai. He sees it as offensive.
There are a few others, but Odin is the most-serious remaining contender. Some might balk at using a language that’s largely dependent on a single person. While not necessarily ideal in the abstract, it provides a focused vision and implementation that
So that’s the next step. I’ll prototype the next interation of Shimlang in Odin.