The awesome power of type introspection

The following post is mirrored on my blog.

I have been making progress on several fronts with Cakelisp and GameLib, but in this post I wanted to focus on an exciting work-in-progress addition I have: type introspection.

What it is

Type introspection refers to the ability to inspect types at runtime. Reflection is the ability to both inspect and modify types at runtime, which I'm not going to talk about here. By types, I'm referring primarily to structured data, e.g. struct in C.

What this means is that given a type my-struct, you could write a loop which iterates over the fields in that struct at runtime and know some information about that field, without having to write a single line of code per each field.

I am going to attempt to convince you why this is an essential feature to have in your arsenal.

Applications

As soon as you can programmatically iterate over the fields of a type, many doors are opened.

Serialization

The first application for type introspection is perhaps the most obvious. The type metadata is used at runtime to serialize the struct to and from various formats. This eliminates a huge amount of error-prone, repetitive, and just plain boring-to-write serialization code.

For example, with introspection support I can write the following:

  ;; Define a struct which is introspectable
  (def-introspect-struct my-struct
    name (* (const char))
    value int
    decimal float
    bad ([] 3 float) (ignore) ;; Don't try to introspect this field - ignore it
    truthy bool
    charry char)

  ;; Create an instance of the struct, for testing
  (var a my-struct (array "Test struct" 42 -0.33f (array 1.f 2.f 3.f) false 'a'))

  ;; Write out the struct, passing in the metadata for introspection
  (write-introspect-struct-plaintext (addr my-struct--metadata) (addr a) stderr)

;; Create an instance of the struct, for testing (var a my-struct (array "Test struct" 42 -0.33f (array 1.f 2.f 3.f) false 'a'))

;; Write out the struct, passing in the metadata for introspection (write-introspect-struct-plaintext (addr my-struct--metadata) (addr a) stderr) [/code]

This writes the following text to stderr (but could just as easily write to a file):

  (my-struct :name "Test struct" :value 42 :decimal -0.330000 :truthy false :charry 0)

I chose a Lisp-style format for the plain text, but XML, JSON, etc. could easily be generated instead.

Plain-text is a great initial serialization format because you can easily read and edit it with a regular text editor. This means you now have data or configuration files. All you need to do is def-introspect-struct the configuration struct, then e.g.:

  (var configuration my-config (array 0))
  (read-introspect-struct-plaintext (addr my-config--metadata) (addr configuration)
                                      (fopen "config.cakedata" "r"))

Schemas

Once you have metadata for a type, there's no reason why you can't use that same functionality on the metadata itself. If you do this, you can then serialize your metadata and export it to other programs. This is essential when writing editors or other programs which cannot have knowledge of the original type (because it is in a different programming language, for example). You could also generate documentation for the types, automatically create a suitable SQL table to store data of the type, or generate e.g. a REST API specification.

Debugging and hot-reloading

Reading and writing structs at runtime is a valuable debugging tool as well. You can print structs to stderr to inspect or log their values, which is much easier than writing a printf statement for structs with many fields.

You could write runtime structs to a file, then watch that file for changes, and hot-reload the file with the new struct values!

Taken a step further, you could provide a custom interface for viewing and editing runtime data and configuration. I've seen this in action at a previous job, where their MMO shard consisted of a dozen or more different servers. They provided the interface over HTTP, so you could access any shard's server via URL in any web browser. When things went wrong you could inspect and even modify values in the web browser, then see the change in-game without having to restart anything. Isn't that incredible?

Editors

On any major game project in development, there comes the problem of how people without knowledge of programming input data into the game (many artists and designers, for example). The solution is commonly to create custom editors for the data. This creates a large amount of work for game programmers because these editors must be created in-house.

Off-the-shelf editors do not have enough context to properly serve AAA game developers' needs. For example, a drop-down field which provides all possible values of an enumeration greatly reduces data entry errors and increases productivity, but no off-the-shelf editor knows the values of your enumerations without being told.

So, we are stuck with implementing our own editors. Compare this pseudo-code:

  (defun power-definition-editor (the-power power-definition)
   (create-text-field "Name: " (field the-power name))
   (create-text-field "Description: " (field the-power description))
   (create-float-slider "Damage: " (field the-power damage) -100.f 100.f)
   ;; etc...
   )

  (defun enemy-definition-editor (the-enemy enemy-definition)
   (create-text-field "Name: " (field the-enemy name))
   (create-text-field "Model Name: " (field the-enemy model-name))
   (create-float-slider "Health: " (field the-enemy health) 0.f 100000.f)
   ;; etc...
   )
  ;; Continues, for every data definition you ever want to have an editor for

(defun enemy-definition-editor (the-enemy enemy-definition) (create-text-field "Name: " (field the-enemy name)) (create-text-field "Model Name: " (field the-enemy model-name)) (create-float-slider "Health: " (field the-enemy health) 0.f 100000.f) ;; etc... ) ;; Continues, for every data definition you ever want to have an editor for [/code]

...to this:

  (defun create-generic-editor (metadata type-metadata editing-struct (* void))
    (for-each-field-in-metadata metadata current-field
      (when (= field-type-string (field current-field type))
       (create-text-field (field current-field name)
                          (+ editing-struct (field current-field offset))))
      (when (= field-type-float (field current-field type))
       (create-float-slider (field current-field name)
                            (+ editing-struct (field current-field offset))))))

  ;; Then:
  (create-generic-editor power-definition-metadata the-power)
  (create-generic-editor enemy-definition-metadata the-enemy)
  ;; We can even load the metadata at runtime and create the editor based on that:
  (create-generic-editor (read-introspect-struct-plaintext "TheWorld.metadata") the-world))

;; Then: (create-generic-editor power-definition-metadata the-power) (create-generic-editor enemy-definition-metadata the-enemy) ;; We can even load the metadata at runtime and create the editor based on that: (create-generic-editor (read-introspect-struct-plaintext "TheWorld.metadata") the-world)) [/code]

Which would you rather spend your precious life time typing? The straight-line version may be simple for a few editors, but quickly becomes more complex once you must maintain dozens of editors, all of which could have several bugs.

The introspection-powered version not only saves you typing every time a field is added, renamed or removed. It also reduces bugs by centralizing all the complexity to one function rather than distributing it to each editor.

If your metadata format supports tags, you can add a tag like no-edit to hide a field. You can also decide to have create-editor check fields against a table of override functions, then defer creation of editor widgets to an override. For example, a color picker widget could be selected for ([] 3 float) rather than displaying three unintuitive sliders.

An added benefit is that the code which edits the structure is completely separated from knowing the actual type of the structure. This means you could write a generic editor which can understand structs it was not compiled to understand, simply by passing the metadata for that struct. You could even write the editor in another language, if you so choose.

You can see an example of introspection for editors taken to an extreme in Unreal Engine. New structs/classes declared with UCLASS or USTRUCT prefixes automatically create editor interfaces through type introspection and UPROPERTY annotations.

Remote procedure calls

Once you have a system for easily serializing types, creating a remote-procedure call system is much easier. When an RPC is initiated, the system serializes all the arguments into a packet along with the procedure name to call. On the other side of the connection, the arguments are deserialized, the procedure is found from a function table, and then it is called.

You will likely want to use code generation for doing the argument unwrapping and forwarding to the procedure.

How it can be implemented

Some languages, like Python, provide type introspection as a built-in feature of the language. C, for example, does not provide type introspection. There are several approaches you can use to add type introspection to any language:

Code parsing: In a stage before compiling your native language files, a separate executable parses the native language files for type definitions and generates type metadata. This generated metadata can either be stored in a custom format and loaded at runtime, or output to a new native language file (code generation) and compiled with the rest of the files. This technique is used by Unreal Engine to add various features on top of C++[0], for example
Compiler modification: A modified version of the language's compiler could be created that understands your custom annotations and generates the type metadata. The disadvantage is complexity (maintaining a compiler) as well as portability[1]
Separate schema: Rather than defining the types in the native language (e.g. C), define schemas in a format that is easy to parse. You then use a code generator to generate the native language type definitions as well as the metadata for the type. This is the approach used by Naughty Dog (convention link) and Google's FlatBuffers, for example. The disadvantage of this approach is the disconnect between the actual implementation - you now have a "special language" that you must write in rather than what you (and your teammates) are familiar with

The big advantage of implementing type introspection on your own is that you can customize it to your needs, whether they be specialized type declarations, different performance characteristics, et cetera.

How introspection is implemented in Cakelisp

Cakelisp has both full-power macros and arbitrary compile-time code generation and scanning, which make type introspection implementable in user-space. That is to say, it is possible to implement type introspection without having any prior support for it in Cakelisp itself.

My work-in-progress implementation of introspection is in GameLib. I added introspection to Cakelisp by creating a macro def-introspect-struct which follows regular defstruct syntax, but allows optional tags to be added to customize the metadata. Metadata is code-generated to the file by the same macro so it can be easily used at runtime. The generated C++ for my-struct, defined previously, looks like this:

  static MetadataField myStructMetadataFields[] = {
      {"name", serializationTypeString, offsetof(MyStruct, name), NULL, NULL},
      {"value", serializationTypeInt, offsetof(MyStruct, value), NULL, NULL},
      {"decimal", serializationTypeFloat, offsetof(MyStruct, decimal), NULL, NULL},
      {"truthy", serializationTypeBool, offsetof(MyStruct, truthy), NULL, NULL},
      {"charry", serializationTypeChar, offsetof(MyStruct, charry), NULL, NULL}};

  static MetadataStruct myStructMetadata = {
      "my-struct", myStructMetadataFields,
      (sizeof(myStructMetadataFields) / sizeof(myStructMetadataFields))};

static MetadataStruct myStructMetadata = { "my-struct", myStructMetadataFields, (sizeof(myStructMetadataFields) / sizeof(myStructMetadataFields[0]))}; [/code]

This data is never edited by hand, and automatically updates whenever the definition to my-struct updates. I will add more types, flags, and fields to the metadata format as necessary.

Conclusion

Type introspection is a great way to:

Reduce the amount of code you need to write
Reduce the amount of errors in your code
Make features drastically easier to implement that would otherwise be infeasible

Next time you are copy-pasting code or dealing with boilerplate after adding a new field to a structure, consider whether type introspection could be applicable.

[0] Parsing C++ is extremely complicated, so you can decide either to support a subset of C++ when defining types or bring in a beastly dependency like libclang to help. A primary factor of my starting Cakelisp was how frustrated I was with C++'s parsing complexity. The syntax effectively encrypts the type and function signatures I need to do things like introspection. [1] Some platforms, especially game consoles, require you to use their compiler, which they themselves modified, and likely will not release source code of.

The awesome power of type introspection

Comments