Metaprogrammable, hot-reloadable, no-GC language for high perf programs (especially games), with seamless C/C++ interop

About Cakelisp


Screenshot from 2023-02-10 13-50-06.png A work-in-progress voxel game written in Cakelisp.

This is a programming language where I can have my cake and eat it too.

The goal is a metaprogrammable, hot-reloadable, non-garbage-collected language ideal for high performance, iteratively-developed programs (especially games).

It is a transpiler which generates C/C++ from an S-expression syntax. Cakelisp takes some inspiration from Lisp, but is not compatible and does not aspire to become "a Lisp".

I was inspired by Naughty Dog's use of GOAL, GOOL, and Racket/Scheme (on their modern titles). I've also taken several ideas from Jonathan Blow's talks on Jai.

You can also read the introduction to Cakelisp.

The main repository is here.

Features

  • The metaprogramming capabilities of Lisp: True full-power macro support. Macros can use compile-time code execution to conditionally change what is output based on the context of the invocation
  • The performance of C: No heavyweight runtime, boxing/unboxing overhead, etc.
  • "Real" types: Types are identical to C types, e.g. int is 32 bits with no sign bit or anything like other Lisp implementations do
  • No garbage collection: I primarily work on games, which make garbage collection pauses unacceptable. I also think garbage collectors add more complexity than manual management
  • Hot reloading: It should be possible to make modifications to functions and structures at runtime to quickly iterate
  • Truly seamless C and C++ interoperability: No bindings, no wrappers: C/C++ types and functions are as easy to declare and call as they are in C/C++. In order to support this, I've decided to ignore type deduction when possible and instead rely on the C compiler/linker to relay typing errors. Cakelisp will blindly generate what look like C/C++ function calls without knowing if that function actually exists, because the C/C++ compiler will tell us what the answer is
  • Compile-time code modification: After all macros are expanded, the programmer can specify compile-time functions which can do arbitrary modification of the expanded code. This makes it possible to validate functions, automatically insert profiling instrumentation (similar to this Jai demonstration), and other tasks which would be cumbersome or impossible to do with macros alone
  • Output human-readable C/C++ source and header files: This makes it possible to use Cakelisp in a subset of your project. It also means Cakelisp will work on any platform C/C++ works on. Generated code closely resembles the source Cakelisp code whenever possible. Existing C/C++ debuggers will work great on the generated code
  • Build system: Simple projects will automatically be built and linked into an executable. Complex projects can use compile-time code execution to override stages of the build process. The code essentially knows how to build itself!
  • Low dependencies: Cakelisp is made with the C and C++ runtimes, some STL, a separate C++ compiler (provided by your system or your compiler of choice), and some operating system-level dependencies (on Windows, Windows.h; on Linux, libdl and syscalls like exec()). Generally, if you can compile C++ on your system, you have everything you need to build and run Cakelisp. Note that programs written in Cakelisp do not need any of these dependencies, they are only necessary for the Cakelisp executable itself

How is this unique?

Cakelisp is especially unique in its acceptance of the necessity of C (and to a lesser extent, C++). People with existing projects in C will find the seamless interaction with Cakelisp an essential feature. In fact, Cakelisp's build system works so well with C that you can only use Cakelisp for building and compile-time code execution, and keep all your "runtime" code in C.

If you decide Cakelisp is no longer valuable to your project, you can still leverage the human-readable generated C/C++ code which was originally in Cakelisp. This greatly reduces the risk of using a niche language like Cakelisp.

Making a language which outputs machine code/LLVM IR/etc. is a much larger effort than one which outputs C/C++. Cakelisp is a compromise of practicality over "purity". I learned that I was more interested in high-level language features like metaprogramming which could be implemented atop a C base than low-level language design like type systems.

Another benefit of this is that learning Cakelisp is easy for anyone who already knows C. I didn't change things I didn't have good reasons to change.

I wanted to be able to write games in this new language after a couple months of development, not several years. I was able to accomplish that goal by leveraging C as the output format.

In announcing this project I have received many tips about other languages with similar goals. I have a list and some comparisons available here.

What is the current state?

The current version of Cakelisp is quite usable:

  • I have written a simple Rush Hour game atop SDL in Cakelisp as a dogfooding experiment (available here)
  • Cakelisp's build system is powerful enough to replace any build systems I've used on my previous projects (primarily Jam)
  • Cakelisp runs on Windows and Linux, and is known to run on mac OS, though I don't own a Mac for testing. Unlike many open-source projects, Windows support is fully native, i.e. uses Windows APIs instead of something like Mingw, and MSVC is the default compiler for generated code
  • I have integrated Ogre3D (v.2, aka ogre-next), Handmade Math, and SDL as a test of how 3rd-party code feels in Cakelisp. Cakelisp's module system makes it much easier to bring these dependencies into new projects without having to touch any build system

What's in the future?

  • Improved build times. Precompiled headers are a huge hassle in most build systems, but make a big difference in build times (my measurements showed ~40-50% speedups!). I plan on adding features like these to make Cakelisp a bigger time-saver
  • New language features. "Defer" is the big one on my mind. There is work to be done to make hot-reloading more robust
  • Better output language handling. I'm planning separate "strict C" and C++ output modes to best suit your project: you can write the majority of your code to output to C if you like, then have some parts of your project which e.g. interact with C++-only libraries output C++. The more distant future could see language outputs in Rust or Zig, though I don't yet have a good reason to do that personally

My focus on practicality means I'm trying to only add things which emerge as big wins while making the type of projects I'm making. I'm avoiding the "Big Idea" style of language development where ideology dominates practicality. I'm instead making a language where lots of small things add up to make a big difference, especially in regards to developer satisfaction.

Due to this project-driven approach, Cakelisp receives love in between small (month-long) projects. With more interested parties involved, Cakelisp could be updated more often, especially if others start their own projects using it.

Read more
Filters

Recent Activity

I was motivated by the energy of the conference and especially Mason's Vulkan talk, so I wrote up my experience making a bare metal system on Raspberry Pi 4, which included programming GPU shaders in assembly. The interface is not too different from Vulkan, so it should be of interest to people comparing his talk to the command lists at the bare metal level.

I can't figure out how to post this to Handmade Network as a blog...

What's the status of Cakelisp?

I use it every day. It is great for me. I've programmed my watch, my phone, my handmade mechanical keyboard, my current paid contract gig project, and more using Cakelisp.

I wouldn't recommend others use it because they can likely get similar success from creating their own C compile-time code generation/metaprogramming and custom build systems, similar to what Ryan Fleury did with RADDebugger. Taking ownership of your toolset and customizing it to your use case is the lesson to be learned here, not using someone else's exact tools. Different problems require different solutions.

I'm extremely glad I invested in building Cakelisp and continue to reap benefits from it.