Playtech is a leading supplier of online gaming and casino products. As part of the GPAS (Gaming Platform as A Service) Developer Programme Playtech provides game design and development tooling including the Ryota Maths Editor.
Playtech mathematicians and game designers need accurate, detailed game play simulation results to create fun experiences for players.
While software developers have been able to iterate on code in an agile manner for many years, for non-analytical solutions, mathematicians have had to rely on slow CPU-bound Monte-Carlo simulations, waiting, as software engineers once did, many hours or overnight to get the results of their latest changes. These statistics are also required as evidence of game fairness in the highly regulated online gaming business.
Playtech has developed an AWS Lambda Serverless based solution that provides massive burst compute performance that allows game simulations in minutes rather than hours.
Monte-Carlo game simulation is a problem that is easily to parallelize. A typical simulation involves executing one billion independent game-plays and then requires aggregating the game results into a single set of statistical results.
Within Playtech we have two key users of the game simulation function. Game designers who want to tweak and iterate their game design, observing the impact of such changes on game play statistics. Game engineers who need to perform software updates and perform certification builds, validating each time that the game play is correct and generating statistics that can be used in documentation and sent to gaming regulators.
The initial, standard solution to executing a Monte-Carlo game simulation is to use a single server (either on-premises or an Amazon EC2 instance) and run a heavily multi-threaded simulation. A typical simulation might use 12 threads for 8 hours to deliver a simulation result — approximately 350K thread-seconds of execution.
This multi-hour execution time has several very significant downsides for our game designers and game engineers.
- The inability for game designers to iterate a design effectively.
- The inability for game engineers to perform game builds and generate sign-off stats in a timely manner.
- There are even more significant delays if more than one game designer / engineer wants to perform simulations concurrently.
It is also worth noting that one critical constraint is that the game logic code that is being simulated must be the exact same JVM code that we run in our production services. This is important as online gaming is a highly regulated industry and there is strict requirement to be able to show that the simulation results will exactly match those seen by real players in production.
The next challenge is that each simulation run must have deterministic results. It is important that when we re-run a simulation, we can determine whether or not a simulation execution is exactly the same as we would expect when performing maintenance releases.
The final challenge is that game engine development is a constant iterative activity. We thus require a continuous delivery function that will swiftly take game developer’s changes and build and deploy them to make them available for simulation execution in a matter of minutes.
We support many game development teams with many games in simultaneous development. We thus typically see over 100 game code updates a day and these often impact the code in our game engines (And thus our Lambda execution functions). It is also critically important from a QA and regulatory perspective that we know exactly which version of not just every execution component, but every orchestration and statistical analytical component is used on every simulation run.
Monte Carlo Simulation has three distinct components to it:
- Simulation orchestration which controls the highly multi-threaded operations
- Individual sim execution and Individual result generation
- Results reduction
The above process can be thought of as a map-reduce activity with the map being the execution of a single (Or small batch) of sims, with the output of the map being the statistical results from the execution of those sims. The reduce is a reduction operation on the stats output.
The solution that we have developed is to use AWS Lambda for all three of the above operations with S3 to store the intermediate and final output API Gateway to control function execution versioning and Amazon CloudWatch for logging and cost monitoring.
AWS Lambda allows us to scale to thousands of executing threads in seconds and many more in minutes. A simulation that would take 50 hours on a single thread or several hours on a multicore machine can be executed in a few minutes.
It is not however possible to simply kick-off thousands of execution functions and have them write the results to a single location.
The Execution of a single simulation run therefore proceeds in four phases:
Figure 1. Highly Threaded Monte Carlo Simulation Execution Phases.
- Validation: This phase validates that the simulation is likely to complete in a reasonable time. Games with very slow code can cause very large cost impacts (see below).
- Fan Out: A single Initial Lambda function calls out to multiple Fan-Out functions that in turn call more, that eventually call the Simulation Execution functions.
- Simulation Execution: These Lambda Functions execute a small batch of Game Simulations (typically 10,000)
- Results Reduction : The final simulation state is to reduce the simulation output stats using a multi-stage reduction approach.
Read the full blog to learn how Playtech architected the solution on AWS, and lowered costs compared to their on-premises environment.