Greetings, Elyrians!
Welcome to the second installment of our new monthly CoE Development Update! If you're curious about the focus, intended audience, or objectives of these updates, you're invited to review last month's CoE Development Update: May 2023.
Part of this development update will discuss the completed work on integrating an Actor Model into our Entity Component System (ECS) to create our Distributed Entity Component Actor System (DECAS). If you'd like to learn more about what this means and why it's significant, you're welcome to review last month's Dev Journal: May 2023 which provides a high-level introduction to concepts like traditional game loops, ECSs, parallel processing, and our intended hybrid model for the CoE architecture.
However, if these concepts seem too technical, please don't worry. It's perfectly fine to skip the Dev Journal, as the remainder of this development update is designed to be as accessible to everyone as possible, regardless of technical expertise.
With the new format of these Development Updates, we've included a navigation tree for your convenience. Feel free to use this tool to navigate directly to the sections that interest you the most. Of course, you're welcome to read the update from beginning to end if you prefer. Thank you again for your time and interest, and let's dive into this month's update!
Continuing with our usual routine, let's review the high-level scope outline for CoE, followed by a deeper dive into our current primary focus of development.
If you look at the image above, it illustrates the different parts needed to bring Chronicles of Elyria to life. These pieces include creating all game assets, finalizing the functional and narrative design, the technical engineering, the gameplay mechanics, and lastly, using our world-generation tools to fill each starting world.
Currently, my focus is centered entirely on CoE's Engineering. To break it down further, apart from the various game mechanics, the game's engineering is divided into several parts, such as the Server-Side Game Engine, the Server Platform, and a few others. As of now, I'm pleased to share that I'm almost done with the Server-Side Game Engine.
When we look closer at the Server-Side Game Engine, you'll notice some tasks still left on the docket. These include fine-tuning the AI, finishing up a handful of services, implementing a final logging solution compatible with our platform, and developing a serialization and storage model that our game client and services can quickly access in the event of a service interruption.
However, the cornerstone of our Server-Side Game Engine is the Distributed Entity Component Actor System. I'm thrilled to announce that, after the past month's efforts, it's nearly finished. The remaining work to do there, such as the Repository Synchronization, Singleton Entity Service, and Configuration, can only be put to use once the Server Platform is up and running and the game engine is being distributed across multiple processes. The only other element on the list is the Fluent-Based View Queries which will allow me to expand the ECS Systems in new ways and create far more interesting systems. We'll talk more about that in another Dev Blog at some point in the future.
As highlighted in the previous Development Update and extensively in the recent Dev Journal, the past month's focus has been on finalizing and integrating the Actor Model into our ECS. In addition to that, my focus on parallel execution also led to work on the Execution Stage Scheduler. Keep reading below for a more detailed rundown of this month's achievements.
To achieve the ambitious scalability requirements of Chronicles of Elyria - hosting an entire game world - it's crucial that the systems running the world are distributed across multiple processes. These processes function asynchronously, accepting inputs, updating the game state, and, when necessary, delivering responses.
In the last Development Update and Dev Blog, I discussed how transforming the Soulborn Engine from an initially synchronous architecture to an asynchronous one was necessary to operate in such a distributed environment. And now, as illustrated in the scope outline above, that task is accomplished! Over the past month, I've completed integrating the Actor Model into the Soulborn Engine, and I'm poised to begin reworking our network stack. CoE will return as a full-fledged MMO in no time, now boasting more features and functionality than ever.
One of the hurdles encountered while integrating the Actor Model into Medium's systems was realizing that not only the systems and component repositories needed to be thread-safe. Any shared resource used by those systems also had to be thread-safe. This presented an initial challenge in integrating the Actor Model into Medium, as the most crucial part of the movement and collision system - the Spatial Partitioning System - needed to be thread-safe.
I previously implemented a caching strategy to boost the performance of the spatial partitioning system, which divides the world into manageable segments. However, this strategy prevented multiple queries from being performed on the system simultaneously. This doesn't align with our goal of supporting millions of entities worldwide. Therefore, besides integrating the actor model, I also had to spend some time this past month ensuring that the octree became thread-safe.
With all that completed, you can see that with the octree thread-safe and the actor model integrated, Medium now runs better than ever!
In the video shared above, you'll notice that on a 1km x 1km map, we can now support 40,000 entities. All of these entities are moving, processing collisions, and observing functional gravity, maintaining a steady frame-rate of around 32 FPS. Comparing that to the final video from May, you'll note a significant, 100% increase in entity capacity from May's 20,000 entities at the same frame-rate.
You can see from the performance counters I added to the Aura, Collision, and Movement systems that about 90% of the frame-rate is consumed by the game's update loop, with less than 10% allocated to the render/draw phase.
Likewise, the spatial partitioning system currently uses a full 65% of the game loop time. Initially, I assumed that intersection and related queries would be the primary time sinks. However, they only account for less than 20% of the frame's time. The vast majority is in updating the octree itself each update, as it stores multiple references to various branches, along with the entities themselves. Thus, adding and updating entities can be time-consuming, particularly when there are 40,000 entities to update.
While these numbers seem "okay," the critical question is - are they satisfactory? To shed more light on this, let's look at something I've been saying since 2016, for which I found evidence in 2019 in the Official CoE Discord's community channel.
The quote I'm sharing next was my response to a community member's query on CoE's scalability potential, particularly toward our target playable character count of 250k. Have a look.
As I pointed out then, and it still holds true now, you can segment your world into multiple processes when your world is vast enough. Each process then manages the state, collision detection, and so forth for a considerably smaller portion of the world. This approach is similar to the spatial partitioning mentioned earlier, but rather than using spatial partitioning algorithms like octrees, quadtrees, and so on to cut down on calculations, you divide up whose responsibility it is to perform those calculations. Then, when an entity moves from one zone or cell to another, you migrate the management of that entity from one process to another, possibly residing on a completely different machine.
By adopting this strategy, you can continue to subdivide the world until, eventually, the CPU processing power ceases to be the bottleneck and, instead, the challenge becomes network bandwidth and latency. Each node must communicate with its neighboring nodes to prevent popping and other artifacts. They need to "share" entity state information over a specific area.
So, as I emphasized in the quote, the real challenge isn't hosting 250k characters in a single world. It's about supporting 250 players updating simultaneously within an area the size of a large football field or gymnasium. While this might have been an exaggeration, it is true that the real hurdle is in enhancing the performance of a single node that can no longer be reasonably subdivided to the point where it can support all intended game mechanics for as many players as can fit in that space.
Keeping this in mind, I've prepared the following video for you.
You'll quickly observe that the area in this video is significantly smaller than the 1km x 1km landscape you saw earlier. Indeed, it measures just 128m x 128m, slightly larger than a large sports field in each direction.
The video starts modestly, with the addition of 250 moving entities to the world. I progressively add 250 more every few seconds until we reach 1,000 moving, updating entities and are still operating smoothly at 60 FPS. To further push the limits, I introduce another 10,000 entities! Even at this stage, we still hold a steady 60 FPS, so I add yet another 10,000! Now we have 21,000 entities inhabiting an area of 128m x 128m. To prove that we're not just handling movement and collision, I enable the terrain to show that we're also processing gravity.
By this point in the video, the entities are so densely populated that only a few white cubes are left on the map. These white cubes aren't colliding with anything at the moment.
To conclude, I add a few "buildings" to the map, each with auras, to demonstrate that the aura system still functions optimally. The "buildings" are yellow, and all entities within their auras are colored orange.
So, what we have here is 21,000 entities, each detecting collision against each other, the terrain, and the building auras, in an area so dense that there's only a handful of entities not colliding, on a 128m x 128m terrain, operating at almost 40 FPS.
Sure, I can optimize the engine further. But is it robust enough to handle the large volume of entities (and players in particular) we're planning for CoE? Absolutely. It's ready.
With the DECS near completion, my focus for the upcoming month will shift toward the Server Platform. While initially, KoE might not emerge as a fully-fledged MMO Grand Strategy Game, it's crucial to keep in mind that much like the necessary integration of an Actor Model, having CoE/KoE operating on a distributed server architecture from the get-go ensures a smooth transition from this point to the end of development.
Allow me to illustrate this with a construction analogy. The Game Engine I've been developing thus far is akin to a building's foundation. While it may not visually impress, that's not its purpose. It's meant to be robust and resilient and provide a structure to build upon. Similarly, the game engine itself serves the same function. While there's little to showcase during its development, the focus lies in establishing a solid base on which the future games of Soulbound Studios will be built.
Continuing the analogy, if the Game Engine is the foundation, the network platform resembles the heavy machinery involved in construction, like the cranes. These are only essential once you're aiming for scale. However, it's crucial to ensure they function correctly with the materials at hand and that the team is well-versed in their operation and capabilities.
Focusing on developing the networking layer now and getting the platform to host the MMO means we can validate that our scaling tools work harmoniously with our construction.
That concludes this update! I am, as always, grateful to our supporters and fans who consistently send words of encouragement via the Official CoE Discord Server. While I may only sometimes be able to respond promptly, I do read every single message I receive and ensure to reply when time permits.
And yes, I occasionally receive messages from individuals advising me to give up and abandon the project. But I can't - even if I wanted to. It's Soulbound. I will persist until it's done. One stone, one building block, one play test at a time, until the mission is accomplished and the studio's commitments are fulfilled.
Pledged to the Continued Development of the Soulborn Engine and the Chronicles of Elyria,
Caspian
