Shimlang has gone through many iterations since it was first conceived. Originally it was meant to be an interpreter that was small enough to include directly include in a project repository for multiple platforms, and to be deployable as single file executables like Love2D.
While that was a neat concept to play around with, that motivation was fairly thin. I know that I’ve had issues with other projects relying on Ruby or Node or Perl for some tasks, especially if they required newer versions of those languages that weren’t readily available.
Conversely, most Linux distros will already have Python and I can easily write Python that doesn’t depend on third party libraries (eliminating venv issues for others) or newer features (Python 3.6).
That left my primary use-case for a custom language — gamedev.
The biggest inspiration for me is Tomorrow Corporation’s tech demo where they show off very impressive tooling for development, debug, and collaboration. In particular, time-travelling debugging, bug reproducibility, hot reloading, and visual debugging were huge draws for me. They wrote a compiler for a custom C-like language that allows them to build for whatever platform they want to target. However, during development they have an incredible amount of tooling around an interpreter for this language.
One of the most impressive things about this demo is how cohesive the system is. There’s no barrier between scripting and native code. All code is in the same custom language. I don’t want to use a low level language for gameplay programming, so I know this solution isn’t for me.
Shimlang is a game scripting language with a strong focus on debuggability, determinism, modern syntax, and stable performance. All memory for the interpreter is stored in a contiguous chunk of memory that can be indexed by 24-bit addresses, 134 MB in total. The interpreter natively operates on 64-bit values. With NaN-boxing and a 4-bit tag, these 64-bit values can store two 24-bit addresses. Since all of these values store u24 addresses (rather than pointers), serialization and determinism become easier to implement.
Garbage collection is handled by a completely separate thread. All memory writes set a dirty bit for their block of memory. The GC thread keeps a copy of the entire interpreter memory, and updates its copy and clears these bits as it goes along. GC happens on this copy, and runs concurrently with interpreter execution. There are no frame-hitches from stop-the-world garbage collection. While it might be possible to implement compaction, it’s not clear how to do that without pausing the interpreter. There’s also the option of writing an incremental GC, but that complicates the write barrier and seems like it would involve a lot of synchronization between the mutator and GC threads.