In our latest Livestream, we talked with Peter Burzynski of Midwinter about flexible design and offloading - watch it below.
Our last Livestream was an overview of the latest version of SpatialOS for Unreal 2019.1. This time around, we went over one of our previous topics by taking a deep dive into how the SpatialOS GDK for Unreal unlocks more flexible design and computational offloading.
We were also lucky enough to be joined by Peter Burzynski, Director of Engineering at Midwinter Entertainment, who talked to us about his team are using offloading for their game Scavengers.
Traditionally, when you build the architecture for your online game, you have a single server doing all the computation - AI behaviour, player movement, network connections, everything. This is fine for 64 players with a handful of AI, or just 100 players and no AI. Go past that, and the server gets overloaded, FPS drops and you have a bad experience for your players. It’s exactly the kind of problem that SpatialOS engineers love to solve.
Using SpatialOS for Unreal 2019.1, you can offload the computation for classes of Actors, such as all AI actors, from your main gameplay server-worker to a separate Unreal server-worker. On the left of this diagram is a traditional server using Unreal’s native networking to connect to any clients. On the right, two (or more) Unreal servers using the SpatialOS Runtime to connect to any clients.
If we look at the right-hand diagram in greater detail, the SpatialOS Runtime is keeping track of game states, authority and interest - that is, which servers can see and have authority over which actors in the game world. The auxiliary AI server is overseeing NPCs, whilst the main server on the right is controlling the core game loop. This enables you to increase the scale and complexity of the world beyond what a single dedicated Unreal server can achieve.
To verify the performance gains of this computational offloading, we made an Example Project, with a deliberately intensive implementation of AI NPCs and navigation meshes. We then added a large number of these into a game world as Simulated Players, to see what the load was and how we could mitigate it.
As you can see from these graphs, we ran a large number of tests with varied numbers of simulated players and AI NPCs, some using native Unreal single-server networking and some using Unreal with SpatialOS. Above the line in both graphs is where the FPS dropped below 20.
The results seem clear - the area between the two lines is the extra headroom in terms of player numbers and AI NPC numbers that a single offloaded server gives you. For example, using native Unreal single-server networking, a 70- or 80-player game would have to work just as Battle Royales do today - without any AI NPCs. Add SpatialOS, and you can fit 300-400 NPCs in there, utterly changing your design. What would Fortnite look like if the battlefield was swamped by AI wildlife?
To try out offloading, you can use a new Example Project we’ve built called the Crashbot Gym. It’s a simple map with a row of robots with rudimentary artificial intelligence running headlong at a row of hostile turrets. The turrets use the Unreal Engine’s perception system to detect the robots and deactivate them, forcibly.
You can use this to test offloading locally, by adhering to the steps in our Multiserver Offloading tutorial. If you follow this, you’ll be able to split off the crashbots’ AI Actor Group from the turrets’ AI Actor Group and run it on a separate Unreal server.
Scavengers is a multiplayer co-opetition shooter with a rich PvEvP mix and survival elements - it has a large number of players and a large number of AI enemies. The developer, Midwinter Entertainment, is working very closely with our GDK team and has helped in the design of offloading, as Peter Burzynski, Midwinter’s Director of Engineering, explained on the Livestream.
“From the very beginning of the discussions around offloading, we’ve been excited, because it allows us to think about classes of things we can potentially move off our game server and reserve performance there, effectively. How do we move off latency-tolerant, compute-intensive work? Moving AI off is a good, good example.”
“One of the things we’re really impressed with is just how easy it is to get started. There’s a bunch of easy click-boxes, and I can assign Actor groups easily - the changes you have to make, honestly, are relatively minimal. We already have wrappers so we can have our gameplay systems react to those and we can have code that only runs on the offloaded workers.”
You can find out more about how SpatialOS has helped indie-sized Midwinter create a AAA game in our case study.
We have more plans for offloading. We’re looking at how we handle ‘interest’ - for example if an Actor group can be owned by a worker, how do you define which workers are interested in it? With large groups of objects, being able to define which workers are interested in them and know they exist will really help with performance.
Further down the line, we also want to experiment with how offloading interacts with zoning. Offloading is where you have servers running different types of game system. Zoning is a more complex system where you run multiple identical servers running the same game systems but divide their authority spatially to create larger, seamless game worlds. Zoning is already a feature in SpatialOS for Unity and coming to SpatialOS for Unreal. We want to explore hybrid approaches between zoning and offloading.