There were several cleanup and features added to Cakelisp since the last post:

Precompiled headers for comptime compilation is now merged to master. I put in a lot of work to get them working on Windows as well, though I didn't see the performance improvements I was hoping for on that platform.

tokenize-push was rewritten. This generator is the primary way to populate new Token arrays, and is the foundation of all macros in Cakelisp. For example: 
1
2
3
4
5
(defmacro array-size (array-token symbol)
  (tokenize-push output
    (/ (sizeof (token-splice array-token))
       (sizeof (at 0 (token-splice array-token)))))
  (return true))

This example macro uses tokenize-push to add an array-size function to the output array. This is approximately like the C preprocessor except that tokenize-push works on the token level rather than the individual character level.

My first implementation of tokenize-push was dreadful, but got the job done at the time. It would generate code to create the tokens which looked like this: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
std::vector<Token>& outputEvalHandle_0 = output;
if (!tokenizeLinePrintError("(/  (sizeof ", "Dependencies/cakelisp/runtime/Macros.cake", 9,
                            outputEvalHandle_0))
{
	return false;
}
PushBackTokenExpression(outputEvalHandle_0, arrayToken);
if (!tokenizeLinePrintError(") (sizeof  (at 0 ", "Dependencies/cakelisp/runtime/Macros.cake", 9,
                            outputEvalHandle_0))
{
	return false;
}
PushBackTokenExpression(outputEvalHandle_0, arrayToken);
if (!tokenizeLinePrintError(")))", "Dependencies/cakelisp/runtime/Macros.cake", 10,
                            outputEvalHandle_0))
{
	return false;
}
return true;


As you can see, this calls the tokenizer on a static string passed in as an argument, which means the tokens that were already parsed when the macro definition was originally evaluated must be re-tokenized and allocated.

This rewrite accomplished many goals by solving all the limitations of tokenize-push. The new version reuses the tokens already in memory to output tokens, which gives the following benefits:
  • Token sources (line and column numbers) are more accurate
  • Macros are unlimited in length (previously, there was a max of 1024 unspliced characters, which I ended up hitting)
  • Strings no longer need extra delimiting, which was a source of frustration and bugs
  • Performance is better for several reasons: no more tokenization from strings, greatly reduced macro compilation time (fewer symbols in each macro's generated .cpp file)


It works by saving a pointer to the tokens in the ObjectDefinition. This pointer is later retrieved by the macro at runtime via the definition name and tokens CRC. I decided to use a CRC because it was the only stable identifier I could think of. The following were considered and rejected as identifiers:
  • File/line number. This causes unnecessary recompilation if the file the macro definition is in is modified
  • Incrementing counter. This doesn't work because macros within macros can cause non-sequential incrementation due to unpredictable macro resolve/evaluates


While this system is better in every other way, it does require all macros be evaluated each time cakelisp is run in order to have the tokens loaded. This may be an issue if incremental compilation is ever introduced, because now the macro's tokens need to be loaded by some separate system. The old tokenize-push generated source files were completely self-contained.

Here's what the generated code looks like now: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
std::vector<Token>& outputEvalHandle_0 = output;
{
	TokenizePushContext spliceContext;
	TokenizePushSpliceTokenExpression(&spliceContext, arrayToken);
	TokenizePushSpliceTokenExpression(&spliceContext, arrayToken);
	if (!TokenizePushExecute(environment, "array-size", 1677447134, &spliceContext,
	                         outputEvalHandle_0))
	{
		return false;
	}
}
return true;


TokenizePushExecute() traverses the tokenize-push token list stored under the CRC 1677447134 on the "array-size" definition and pops expressions from the spliceContext based on the interpretation.

Cakelisp can now be executed within another Cakelisp's comptime.

This was a feature I picked up from Jonathan Blow's language, specifically this video, if I recall correctly. Previously, my "RunTests" file, which tests various features of the language, would run Cakelisp in separate subprocesses. This had the drawback that I couldn't easily attach a debugger or run valgrind on individual tests, nor test the overall memory of the system.

Cakelisp is already being exposed to the comptime functions, so I moved some functions around so that creating a new sub-environment and evaluating Cakelisp within that environment was possible.

File Helper

The project I'm working on next using Cakelisp is a file manager application with an interface based on Emacs' find-file. I focused on getting the directory browsing feeling fast first, and planned the next versions. I also did some research on the important file-size visualization techniques which will be File Helper's X-factor.

Unexpected benefits of Cakelisp

I included Cakelisp prominently on my résumé, which got a lot of interest and questions during a recent job interview I did. I think it was a great project to show my skills and interest in programming. It helps my résumé stand out from the rest.