Sync Engine

Team of 4
September 2024 - Current
Skills Utilized: C++, Multithreading, Reflection, Serialization, Profiling & Optimization, Game Engine Programming

Introduction

For my senior year project, I set out to build a fully-fledged game engine. While I had prior experience with game engine development, I had never created a complete engine from the ground up and wanted to take on the challenge. To differentiate this project, I aimed to implement real-time collaborative level editing, inspired by Unreal Engine’s Multi-User Editing feature.

The engine was developed in C++20, utilizing CRTP to elegantly collect type information. This allowed the engine to automatically gather information on all subclasses of Actors and Components. I designed a highly intuitive serialization and reflection system, requiring only a single-line declaration per class while ensuring compatibility with field reordering, addition, removal, and even virtual inheritance. To support fast iteration, I implemented DLL hot reloading for game code. Additionally, I developed an intrusive profiler module using assembly to automatically track execution times for all functions, which helped optimize serialization performance by up to 1300%.

I handled all aspects of the engine except for graphics backend, which were delegated to two graphics programmers, and the networking required for collaboration, which I worked on alongside a network programmer.

Showcase

↑ Overall Demonstration

↑ DLL Hot Reloading Demonstration

Technical Highlights

Separate Graphics Thread

To improve both game logic and rendering performance, I separated the render thread from the game thread and built a dedicated communication layer between them. When registering graphics objects, I implemented a double-buffered approach: two lists were maintained, with the render thread processing one while the game thread added new objects to the other. Once the render thread completed its work, the lists would swap, ensuring that neither thread had to wait on a lock.

Additionally, the editor, built using ImGui, required updates to be processed on the render thread due to graphics constraints. However, modifying actor or component fields directly from ImGui led to data races. To resolve this, I redesigned all value modifications as commands that were stored in a buffer, allowing synchronized access between the game and render threads. This solution later proved invaluable also when implementing real-time collaborative editing.

Reflection

I designed an intuitive reflection system for actors and components using template metaprogramming and macros, making field reflection and serialization effortless. For example, the reflection code for a Transform Component looks like this:

Fields that need to be reflected or serialized are listed once inside the class and once externally. Additionally, editor-specific configurations like specifying Getters/Setters can be done in a single line, keeping the process simple and intuitive.

Serialization

Level data is stored in a binary format while ensuring that field reordering, addition, or removal does not break compatibility with previously saved data. This is achieved by maintaining type metadata for each class. The system stores a hashed name and type ID for each field, allowing it to compare stored field signatures with the current structure at load time to ensure proper deserialization.

↑ Type information structures

Additionally, pointer fields for Actors/Components and Assets are handled seamlessly. Actor and Component pointers store memory offsets, ensuring that objects are reconstructed at the correct locations during loading. Asset pointers, on the other hand, store asset references (paths) instead of raw pointers, preserving data integrity across sessions.

DLL Hot Reload

To enable hot reloading for game code, the engine compiles the game logic into a DLL, which the engine executable loads dynamically. The engine build process produces both an EXE and a LIB, and the game DLL links against this LIB.

A major challenge in this setup was handling global variable ownership between the EXE and DLL. For instance, if the Game class has a global variable and it is defined in a header, both the EXE and DLL end up with their own separate instances. Even using extern and defining it inside the engine’s source files doesn’t solve the problem—since the engine’s LIB is linked to the game DLL, the DLL will still create its own instance.

The solution I devised was to use preprocessor directives to control the definition depending on whether the engine or game is being built:

During the engine build, the header file defines an inline instance. During the game build, do not declare the instance. Then, both for the engine code and the game code, use a pointer to that variable exclusively. This pointer is declared as such using the preprocessing directives:

Engine: extern "C" __declspec(dllexport) inline Game* GameInstance = nullptr;

Game: extern "C" Game* GameInstance;

Using a pointer instead of a direct instance prevents link-time errors in the game build due to missing definitions.

At runtime, the appropriate instance is assigned based on whether the ownership should reside in the engine or the game, and I implemented a helper function to automate this assignment, ensuring clarity and preventing misuse.

↑ List up the global variable to be shared,

↑ And this function will automatically synchronize the pointers.

Intrusive Profiler & Optimization

I developed a profiler that automatically records the execution time of all functions without requiring any additional code from the user. This was achieved by writing _penter and _pexit in assembly and enabling the /GH and /Gh compiler options to ensure these functions are executed at every function entry and exit.

One key challenge was preventing the profiler from recursively profiling itself. To avoid this, the profiler was built as a separate DLL, ensuring its code wouldn’t be instrumented. This required adding EXPORT after the _penter PROC declaration.

↑ Part of the profiler assembly

↑ Example profiling output

Using this profiler, I discovered that directly writing to fstream during serialization was creating a major bottleneck. To address this, I leveraged a custom memory manager to accumulate all data first and write to fstream only once at the end. This optimization resulted in a 1300% performance improvement.

Conclusion

This project allowed me to tackle complex engine systems and refine my problem-solving approach. Designing a reflection-based serialization system that remained compatible across versions, implementing a multi-threaded rendering pipeline, and handling global state in DLL hot reloading were all challenges that required careful consideration.

Beyond technical implementation, the experience reinforced the importance of profiling and optimization, as seen in the significant performance gains achieved in serialization. It also highlighted the necessity of maintainable and scalable design choices, ensuring that features like real-time collaborative editing could be integrated smoothly.

The lessons learned from this project will inform my future work in engine development, particularly in balancing flexibility, performance, and ease of use.