This post is mirrored on my blog.

It has been a while since I talked about my dynamic environment work mentioned in this post inspired by this post. I thought I would give an update, although unfortunately I don't have great news.

My modifications

The approach I settled on for enabling a dynamic environment in a compiled C world was building off of Tiny C Compiler.

I made several changes to TCC for environments, including:

  • Custom allocators for guaranteeing relative reference offsets will be valid
  • Functions to expose ELF symbols so the environment can decide which symbols to re-use from the live environment
  • Automatically convert "local" symbols to be PC relative instead, allowing the environment's existing symbols to be used
  • Add functions for replacing specific symbols with absolute addresses

Difficulties

I wanted to test the environment using a real-world program. For me, that means a graphical SDL2 application using the C runtime. I figured if I could get this working, than most other combinations of libraries would work as well, since SDL2 does link in a lot of different things.

Static linking SDL2 exposed a shortcoming in TCC which caused mysterious crashes. I eventually found the issue and asked the mailing list (archive.org backup), where I learned that the TCC dynamic execution is not used very often. I was able to progress by adding additional validation to the linker and then recompiling SDL2 with position-independent code enabled.

Unfortunately, more issues still remain when I try to static link SDL2. Dynamic linking works better, but still suffers from strange crashes.

Next steps

After working for a few months on this project, I decided I can no longer proceed without a good debugger. Because TCC is compiling straight to memory, conventional debuggers like GDB do not know where to look for debug symbols, and the debugger doesn't know where the linker has assigned addresses for symbols.

I tried making rudimentary callstack printing, memory inspection, and single-stepping, but they just weren't good enough to be useful.

To provide debug capabilities, I have three ways forward:

Assembly level and interrupts

I tried this way first, using the raw assembly instructions for setting hardware breakpoints and enabling single-instruction interrupts. I referred to the AMD64 Architecture Programmer's Manual for these.

The fatal flaw with this approach is that as far as I can tell single-stepping cannot be enabled or disabled while the program is in the interrupt handler for the most recent single-step interrupt. This is because the hardware toggles the single-step trap flag off during the interrupt, then automatically sets it back to what it was before the interrupt. If it didn't do this, the single-step interrupt handler would also single-step, causing an infinite recursion.

I wanted to use this strategy because it's very simple and processor dependent, not operating-system dependent. However, due to this flags issue, I don't think it is possible especially because the operating system gets to put its interrupt handler first, which enforces the flags reset.

This eliminated the possibility of a program debugging itself through its own interrupt handlers. Again, this should be possible at a hardware level, but the operating system ties our hands.

GDB plug-in

The next possible method is to create a GDB JIT Compilation Interface which would allow the environment to register its runtime-compiled code to GDB.

The main drawback to this is a dependency on GDB, which is a large, complicated program. I want to ship my software on Windows as well, which GDB is not the best debugger for. In addition, my end goal is a program which can self-modify and self-debug, and integrating/shipping GDB I expect is challenging. My evidence for that is the distinct lack of really solid GDB-based debugger UIs, despite many attempts.

PTrace debugger

The final option I am aware of is to use ptrace, the interface to process-level debugging on Linux. This has similar issues to GDB in that it is dependent on a Linux environment and I want to also support Windows.

I am also not a huge fan of having to have a separate process running in case the environment hits a debuggable event, but it is necessary on the modern operating systems available. It isn't necessary as far as I can tell at the processor level, but because the operating systems control the lowest level, they force your programs to obey their rules for debugging.

This would essentially be writing a Linux debugger "from scratch", but with some nice advantages like having tight integration with the compiler and linker. Since TCC is so compact and part of the environment, the debugger can leverage it to e.g. compile C immediately rather than having a separate parser for debug commands. It also means the debugger author can modify the linker to make their job easier while working in this environment context, which is not a luxury linker/debugger authors normally have (because they have a wider variety of targets and use-cases to support).

Conclusion

I still think dynamic environments are valuable, but I have another project which has to take priority over this. If someone is interested in moving forwards with the TCC-based environment, please get in touch with me at [email protected].

The goal is to get an environment and debugging experience similar to the Lisp Machines, Smalltalk, etc. without needing to leave behind C and the advantages it has.