SpatialOS Runtime update 2: a new load-balancing system


circuit-desktop blue and purple

Following our last blog post announcing the new SpatialOS Runtime, today we are delighted to reveal the next step: the new load-balancing system. With this Runtime upgrade, you get a much more powerful, configurable and easier-to-understand system, as well as a new “replace-a-worker” tool to help with experiments and debugging. This is a fundamental step that prepares you for the entity database roll-out, as well as future Runtime v2 updates.

What is the load-balancing system?

The load-balancing system is how SpatialOS distributes the simulation work for a virtual world among workers. When setting up a SpatialOS deployment, you need to select a load-balancing strategy. This strategy describes which workers should be authoritative over which components on which entities and how many server-side workers should exist at any point. The load-balancing system uses this strategy to decide whether crashed server workers should be restarted.

What’s changed?

The major changes with the new load-balancing system include:

  • The method by which you define load-balancing strategies.
  • A new “simulation layers” concept that helps you to better organise the workers and components in your game world.
  • Improvements to load-balancing stability, solving the cases where entities move back and forth across different worker regions.
  • Improvements to the predictability, consistency and comprehensibility of a load-balancing strategy.
  • A new “replace-a-worker” tool.

Load-balancing strategy definition

The first piece of the new system separates out load-balancing into two parts: “intent” and “enforcement”. These concepts can be managed and reasoned about separately, making it much easier to implement different strategies that are specific to your game world.

When you upgrade, you will have immediate access to three out-of-the-box strategies: “hex-grid”, “points of interest”, and “rectangle grid”. You can read more about them here.

Simulation layers

The second change is the introduction of ‘layers’. Layers divide up the components that make up the entities in your world into groups, each of which is managed by a different set of workers. This allows you to have a simpler and cleaner way of designing your world and splitting it up into independent game systems.

For example, suppose you have three layers in your game world: physics, weather and chat. Each of these layers would have a dedicated worker type (Unity server, C# chat worker, …) capable of simulating the components that the layer includes. Each layer would also have an appropriate load-balancing strategy to manage the set of workers simulating it.

For a more detailed explanation of layers, check out the documentation here.

Improved stability of authority

To improve the stability of a component’s authority and address the problem of “authority thrashing”, our third change was to introduce hysteresis to the load-balancing system. This means that, under the new system, an entity can rapidly move back and forth across the boundary between worker regions, with a single worker remaining authoritative over it.

Improved predictability of load-balancing strategies.

Fourthly, the new system is designated to improve predictability of strategies. We’ve removed configuration options that the old load balancer could interpret ambiguously. We have also removed support for the old dynamic load-balancing strategy, as it was often misused and configured in ways that were highly volatile and led to inconsistency between what should have been very similar deployments.

In the future, we plan to deliver a new off-the-shelf dynamic strategy. Our aim is that this will be a starting point for a growing, evolving world while also being configurable in a controllable way. For the most demanding games and simulations, we will also eventually be enabling you to fully implement your own strategies tailored to your specific needs.

Replace-a-worker tool

Lastly, we are adding a new load-balancing feature, which we call “replace-a-worker” (see the documentation for local and cloud deployments). The tool allows you to replace a specific server worker with one that is running on your own machine.

You can use this tool to experiment with and debug server workers. For example, you could connect a worker with additional profiling enabled to track down an issue that the server worker is experiencing, or you could connect a worker that implements an entirely different logic, testing out how it would affect your gameplay.

What’s next for the Runtime?

Upgrading to the new load-balancing system instantly enables you to benefit from the “replace-a-worker” tool and the other improvements presented above.

More importantly, it also is a fundamental step towards the next and final piece of the Runtime v2 rollout: our new entity database. We expect to roll out this last step in approximately one month. Soon after that, we will make the new Runtime default for all deployments.

We highly recommend all users upgrade to the new load-balancer at launch. You’ll be able to take advantage of all these important improvements from day one as well as benefit from our ongoing work. Click on the button below to upgrade today.

To find out more, watch this blog: in the meantime, you can ask questions and provide feedback on the forums.

Get started now

More solutions. More possibilities.

Explore all IMS solutions