The subject of reflection has been an important topic for game engine developers in recent years. In the past, having this information registered someplace might have been considered wasteful, but any more it’s almost expected in some ways. If you’re unfamiliar with reflection, the most common example of something a reflection library can do is construct types by stringized names. You could imagine extending this useful feature into many things, like dynamically serializing and de-serializing a type without explicitly writing serialization functions. Reflection allows you to do a lot of things, but to be honest – it doesn’t really let you do anything that you wouldn’t normally be able to do by writing a lot of code manually. You don’t need reflection – but it sure as heck helps.

This post discusses the first third of my work on a side-project to build an automated C++ reflection system.

RefN (Reflection eNgine)

Before I begin, let me formally state what my latest project is.

RefN is a native, unobtrusive, compiled reflection system. Utilizing Clang as a code-generation front-end for reflection registration, it allows developers to enable rich, efficient, and configurable runtime reflection information with very little boilerplate.

Let’s understand the key points in more detail:

• Native – Simple enough, it is a reflection system for a native language.
• Unobtrusive – The library should require little boilerplate code to begin utilizing reflection information.
• Compiled – The reflection information should be compiled into the resultant binaries (or associated libraries).

The target user for RefN is someone who is developing a product which requires more dynamic control over types in some way. It is not recommended (in my opinion ever, but especially with RefN) that you rely on reflection information when you are creating a general-purpose low-level library (physics, audio, input, etc). Reflection is rarely needed, and I worked on this out of a passion to gain a better understanding of the C++ language – which this project has delivered upon immensely. (Before my gamedev friends yell at me for this – I mean in the grand scheme of programming reflection is rarely needed. It has many uses in game engines, and prevents us from having to write gobs of code.)

Being a fan of films, the name is also a little tribute to Nicolas Winding Refn, who directs very self-aware films with a lot of “meta” to the writing and camera work.

If you’re interested in trying RefN, you are free to check the code out and play around with it, I welcome all feedback!

Some of the implemented functionality and pipeline might differ slightly from what is mentioned in these blog posts, this project is living and changes depending on needs. I do ask that you read the following documentation and understand; RefN (in it’s current state) should not be taken as a dependency, and only about a third of the planned functionality is implemented at this time – also I currently only test on Ubuntu 15.10 with g++ 5.2.1 – you have been warned!

• Documentation (see: Getting Started)
• Symbol Aliases (important if you want to play around with RefN)
• GitHub Repo

Design Pillars

So what would reflection look like in C++?

This is such a simple question, but it has a complicated answer. For C++, there are many ways you could implement reflection, which do you choose? I decided to place a premium on: correctness, efficiency, and functionality (in that order). Let’s discuss the design pillars a little more;

1. RefN should strive to be correct above all-else.

This should go without saying, but it’s important to call it out in the design pillars. If the reflection system produces an incorrect result for a seemingly well-defined type attribute: it’s wrong. Fundamentally, as a design philosophy, when selecting between correct and efficient we should choose correct. This gives us a very clear idea of where our priorities lie, and what to do when we come across a feature which straddles the boundary between correct and efficient (which happens more than you might expect).

2. RefN should be all about speed and efficiency.

This is kind of a no-brainer, the big thing that C++ has going for it is speed. There aren’t safeguards, there aren’t sanity-checks, there is undefined behavior. I’m not saying to purposefully add or remove any of these things to the code, but I am saying that the level of safety and checking of a reflection object should be determined by the user, not by the API. For example, unless debugging features are enabled, when you access a vector’s .first() element, does the vector make sure there is something to access? Perhaps we could add this sanity checking (depending on compiler flags), but it should only be present if the user requests it, and should not be required for normal functionality.

3. RefN reflection information should be very generalized.

Whens the last time you looked at generalized attributes in C++?
You can add these suckers anywhere. They can exist within namespaces, you can put them on types, variables, functions – hell, they can exist on a for loop! You could put an attribute on a variable which also includes the definition of the type (e.g. [[attrib]] struct [[attrib]] { int [[attrib]] x; int [[attrib]] y; } [[attrib]] variableName, [[attrib]] otherVariable;). It’s madness, and the only reason I can think it exists is because it has to work for even things the standard committee can’t foresee. Reflection, similarly, would have to be just as malleable – we should support the most common and quickest operations first, but functionality should be able to scale to problems we don’t immediately know we need to solve.

These three pillars touch on the main philosophies I mentioned at the start of this section (correctness, efficiency, and functionality). But let’s define a few more to round out our requirements.

4. RefN should only reflect things the user wants reflected

So few developers really need or use reflection. Writing a game engine? Great, you don’t need reflection. Sure it’s cool, and sure it helps a whole lot (that’s why so many people use reflection) – but if you seriously come up to me and tell me you need it? I’m not biting. C++ reflection should be configurable; definitely at the symbol level, maybe through an attribute, possibly through a translation unit, optionally through a “reflection map” (something that spells out specifically what to reflect and how). What gets reflected by default should also be optional; if users don’t care about typedefs, don’t reflect them!

5. RefN reflection information should be compiled into a binary.

There tend to be two trains of thoughts for people writing reflection systems; either you generate dynamic information that can be loaded at runtime to access data via offsets, or you compile code into some binary. I’d be willing to believe that C++ would prefer the latter, compiling reflection information into each .so or .a, and having the linker resolve either at link-time, or pre-main. Since we cannot control the requirements of pre-main, and there is some complexity in how translation units make it into the final binary, we might have to make some sacrifices here.

With RefN, I decided I would keep these key points, and expand upon a few in ways that I really don’t think C++ would. Over the next few posts we’ll talk about the functionality of RefN and why I decided to add that functionality. The beautiful thing about reflection is there is no right way of doing it; I’m aiming to do a fairly complete sweep of reflection information, but if you only need to reflect one thing (say, construction of types), it might only take an afternoon to get that logic working.

Reflection Pipeline

The first step is understanding our process for how we want to generate reflection information. For an automated system it’s important to have a pipeline that you understand and can justify.

A part of being optimized for speed and efficiency also involves the process of generating this reflection information. If it takes an hour to generate reflection information for a complicated process, we’ve already created a tool that no one would want to use. One of the main traps I see people falling down for automated reflection is to just process all the files in a project. This is wasteful, because it’s more likely that the user will know which files contain the types that the user wants to reflect. So let’s have a small shift in our logic;

• RefN compiles header files.

This simple shift in our process almost entirely solves the issue of long reflection times for massive projects. It follows closely to the ideologies behind which C was originally built; compiling several smaller object files (we can do this in parallel), and linking them into one massive binary. The only problem is that those pesky #include directives can re-introduce reflected types that the user might have already generated reflection information for, so let’s redefine our reflection process slightly.

• RefN compiles header files, and generates reflection information for symbols only defined fully within the given file.

Excellent, this means that if we include a file which had reflection information generated for it elsewhere, RefN will not again generate reflection information for it. Now this does introduce some minor issues: what if a type is defined with other includes (such as an enum which #includes a list of enumeration values – this is not uncommon). We’ll add one more slight adjustment to our definition to account for this.

• RefN compiles header files, and generates reflection information for symbols defined fully within the given file, or recursively throughout directly-included files matching user-provided patterns.

A little complicated, but it will work for most cases. This is one of those times we will ask that the user adheres to some general good practices to allow us to make some assumptions about how the code is formatted. By default, RefN will recursively scan files directly included matching “*.inc”. Here is an example of this:

In the above case, RefN will correctly reflect MyEnum and all of the enumeration constants within it. You can imagine other cases of this, such as a user including some file which opens and closes a namespace based on user defines. But what does this do? Why not just compile everything in one file – wouldn’t that be faster? Well, kind of.

If we only had one file to scan, and it included several headers, it would technically work. This is called – or could be identified as – a unity build. The problem with unity builds is that their total build time is faster, but their iterative build time is way slower. Imagine that you are actively developing a program, and you notice that you need to make a small change in a single header. Under normal C++ circumstances, you’ll only have to rebuild the source files which include that file at some level of indirection. However, your reflection utility will now have to re-reflect all of that information, spanning every indirectly included header file. This means that for every change to any header which is indirectly (or directly) included in any of your reflected files, you introduce a required process which takes the same amount of time as when you first reflected everything.

Note: I also believe the above reflection pipeline to be very configurable. If you really wanted to do a unity build, you could! (add a pattern to recursively reflect “*”, and only have one header with all the required reflection headers)

As you can imagine, the process requires a “link” step, which is essentially mapping all of these separate reflection translation units into one reflection map. The full pipeline looks something like this:

The user can decide on a name for any of the intermediates formed by this process, but above you can see the default names of all of the generated files. The generated “*.r.cpp” files are meant to be then compiled by the target compiler of the user’s choice and linked into the final archive or application. The reflection map is utilized as such:

It may seem like magic, but all that’s happening here is Module.r defines a file that looks like this:

And each of those FileN.hpp.r.cpp files defines the functionality and storage for file1_hpp, file2_hpp, etc. – which in-turn contains all of the logic for registering the types discovered within that file. This allows us to keep all the compiling to separate translation units, and allow us an easy way of registering types. In the future, instead of using human-readable names for the translation units, a more scalable option would be: “<filename>_<full path hash>” in order to ensure that no collision takes place for files which may be named the same, but appear in different directories.

Reflection Storage

So the next problem is “where does the type information get stored”?

The answer to that can be solved by promoting “namespaces” to “symbols” – something that isn’t usually considered. We’ll talk about this more in the next blog post where I discuss the currently implemented reflection types. For now, just know that RefN provides a few globals for you, and one of those is a namespace_obj representing the global namespace. What happens when refn::enable() is called with some kind of translation unit is it calls the virtual init(namespace_obj *) function, which is provided a namespace to register into. By default, the global namespace_obj is passed in.

There are a few cases where we create some static non-local pointers to reflection information which will be used for template access (rarely useful, but good for debugging). How this works is by taking advantage of a property of C++ where templates must only resolve to having one linked definition. So in a header, we will declare a template function. In a C++ file, we will specialize-and-force-instantiate a body for the template. One way to think about this is the template in the header dynamically generates declarations, while the specialization and instantiation in the body generates a specific definition. If you haven’t properly linked with the reflection library, calls to refn::get<MyType>() will actually fail at link-time.

Here is a simplified version of what RefN does:

In terms of runtime, this means that accessing refn::get<MyType>() before first calling refn::enable() will fail.

It’s actually a little more complicated than that – the call actually will succeed, but it will return a “null symbol” which will fail if-checks (it’s actually well-defined). refn::type_t is a whole different design philosophy that we will discuss in more detail in the next blog post along with namespaces. All you need to know for now, is that a type_t can be implicitly constructed from a type_obj*, and if that value is null, then it will fail if-checks and is regarded as a “null symbol”. But more on that next time…

Why though, should it be a static non-local?

That’s a much more interesting question – originally I wanted to have RefN not require a library to link with. The only way to can do this and still have globals (like the global namespace) is by creating a static local variable within an inline function. The linker must per-standard only accept one of the definitions, discarding the rest. However, I eventually found that for compilation this felt inefficient – it became hard to hide logic from the user when all of the logic had to be in the headers. After moving everything to a library, I attempted to also move the global namespace object to a static non-local variable, and realized my runtime was measurably more efficient in stress tests. What happened?

Well, the C++11 standard introduced a requirement for non-trivial static locals – this requirement can be thought of as a some kind of concurrency lock upon initialization (and potentially access) of the type. What the standard states about the initialization of a non-trivial static local objects is the following:

[stmt.dcl] 6.7.4
“[…] Otherwise such a variable is initialized the first time control passes through its declaration; such a variable is considered initialized upon the completion of its initialization. If the initialization exits by throwing an exception, the initialization is not complete, so it will be tried again the next time control enters the declaration. If control enters the declaration concurrently while the variable is being initialized, the concurrent execution shall wait for completion of the initialization.“

These requirements are a bit funny, at the very least we’re looking at some kind of atomic access to a static object to see if the type has already been initialized. If not, the type will probably acquire some kind of lock and then construction of the type if needed. Keep in mind this is a theoretical implementation, I’m simply guessing – but you understand that there has to be some check around these types introducing some kind of efficiency loss (possible implementation: double-checked locking, thanks Andy Hill). The reason you’d ever consider a static local is either if you want to be sure of order-of-initialization, or if you don’t want to introduce a library for linking-in the static non-locals. It’s a hacky trick, and results in less-efficient code – avoid it if necessary.

Summary

There is still a lot to do for RefN, and there is so much more to talk about. I know this isn’t a graphics tutorial, but I hope you guys find this equally interesting. Until I finish this current side-project, I won’t really be focusing on much else. Instead of having a huge period of silence, I figured it was about time to do some blog posts about what I’m working on now. Hope it’s entertaining!

Here is what we learned so far:

• What RefN is, along with it’s design pillars.
• A scalable pipeline for automated reflection.
• How and where we will store reflection information.
• About forced template instantiation, and the efficiency concerns of static locals

Cheers!