#RL-BRAWL

The game runs directly in browsers (about 17 MB to load). Click on the window to play or watch bots, and you can play with either a keyboard or a controller! Unfortunately, on phones and tablets you can only spectate (Bot vs Bot or replays) but not play because I did not make touch screen controls...

the game

The game has three main components: the core gameplay which is a relatively lightweight C++ engine, then we have the Godot engine used for visual rendering inputs and menus, and finally the Python layer that handles the training of the agents. The game is designed to be fast and efficient, allowing for rapid self-play and training of reinforcement learning agents.

The core mechanics are not original, and are based on the frame data (amount of frames each action takes like how long recovery lasts for instance). And specifically it’s the data from Mario in Smash. The stage is also the same as the classic Battlefield stage. It was important for the game to be balanced otherwise my RL agents would abuse broken unbalanced mechanics and the game would not be fun to watch or play.

The game is quite easy to pick up, but the hard bots are actually hard so good luck :)

the agents

All the bots that you can spectate or play against are agents that were trained using only reinforcement learning. This means I never explained the rules of the game to them, they don’t rely on heuristics or reading your input to play. To make it even more fair, the bot operates with a 0.1 second delay (so it cannot react to your inputs instantly). It also has the exact same input delay as a human player, so it is quite a fair fight mechanically speaking.

Since the agents are trained using reinforcement learning (more on that below), they learn to play through a reward signal. During the early stages of training, the reward signal is rich. Here are some events that contribute to the signal: damage dealt minus damage taken, a multiplier for consecutive hits without getting hit back, getting close to the opponent... Each event has a different weight and contributes differently, but overall helps the agent understand what is good and what is bad. The reward is also a zero-sum game: whatever a bot wins, the other bot loses, which leads to this implicit idea of “getting countered is bad” (i.e. if you get hit, you lose points, but I win points, which isn’t the case if we trade hits for instance). Finally, over time this reward signal is made sparser and sparser, and all those small reward components fade to only max 20% of the reward signal. The main reward then becomes winning or losing the game, since it is all that matters in the end.

Another interesting gimmick of the training is that the agents are trained in a self-play manner. As training progresses and the agent gets better against itself, it is forced to learn new strategies to counter itself. Implicitly, what this means is that the agent learns to counter abusive strategies that it has come up with. This is because it will eventually (hopefully) be countered by itself. Another interesting property of the training, is that the agent also plays against previous copies of itself, in order to make sure that it doesn’t forget how to play against old strategies that were “solved”. This mechanic is heavily inspired by the concept of “league training”, and is used in many successful RL projects such as AlphaStar and OpenAI Five. This mechanic is very important, because how embarrassing would it be to have a bot that is good against a high Elo bot but struggles or even loses against a low Elo bot...

Under the hood (PPO)

Each bot is a small neural network called the policy, π_θ(a | s). It reads 173 numbers describing the game (positions, speeds, damage, active hitboxes...) from its own point of view, and outputs a probability for each of 45 actions: every controller combination (stick x and y in{−1, 0, 1} × nothing, jump, light, heavy or dodge). It has two hidden layers of 256 neurons, about 120k parameters. Each output of the network is then mapped to an action in the game. A second network, the critic V(s), learns to predict how well things are going from a given state. That critic is not the bot you are fighting however, the one you fight is the policy.

The reward. At every decision:

A KO is worth ±1. m_t goes from 1 to 1.5 as consecutive hits pile up, and counter rewards counterattacks and whiff punishes. F_t = γΦ(s_t+1) − Φ(s_t) rewards getting closer to the opponent. By shaping the reward function in this way, we make sure that it does not change which strategy is best (does not specify the optimal behavior). The KO, damage and counter terms are zero-sum: what one bot wins, the other loses. On top of that, the extra reward terms fade to 20% of their starting strength over the first half of training, controlled with the parameter α(t), so that in the end what matters is winning.

PPO. Training repeats a loop. First, play 256 games in parallel for 128 decisions each (about 33k samples). Then score each decision by its advantage: how much better it went than the critic expected (GAE):

with γ = 0.995 and λ = 0.95. At 15 decisions per second, that γ means the bot plans about 13 seconds ahead. Finally, update the policy to make better-than-expected actions more likely, but never by too much in one step. This clipping is what makes PPO stable:

An entropy bonus (0.03, lowered to 0.01) pushes the bot to keep trying new things early on.

Self-play. The opponent is a frozen copy of the bot. Once the bot beats that copy in at least 55% of 200+ games, it becomes the new copy, a slightly stronger rung on the ladder. So that it doesn’t forget how to beat older styles, its games are split between: 55% against the current copy, 15% against retired copies, 20% against checkpoints of a previous run (mostly the ones it beats about half the time), and 10% against hand-coded bots (these are the easiest and most predictable).

Numbers. The best bot trained for 480 million decisions on a single gaming PC (RTX 3070), roughly a year of non-stop game time at 60 fps. Ranked against its own past versions, its Elo climbed steadily to 2251, and the difficulty levels in the menu are checkpoints saved during training.

in the browser

The whole game, bots included, is compiled to WebAssembly and runs directly on your machine! The bots are neural networks, but they are so small that they can run in the browser without any problems, and no fancy GPU required. The game is also relatively small, 17 MB (the Godot engine takes up most of this size), which means it’s quick to load unless all of a sudden thousands of people want to check out this page and then my Raspberry Pi server will struggle to keep up with the upload.

The game is served by my backend, and it is a build of the game repo that I export to the website. This means that the window is not a recording of the game, it is the game itself running in your browser.

what’s next

the mechanics and art

The main next steps for this little project are to add more mechanics to be closer to Smash in spirit, and also to improve the art. What shouldn’t take time is adding moves like up+B for more recovery options, more diversity in smash attacks (individual ones for individual characters), and improving the physics because I know the feeling is not crazy good yet.

What will take time is improving the art. I am not an artist, and I am not a pixel artist. I can do some pixel art, but it’s not my forte. Secondly, some of the art comes from royalty free assets websites, some of it made by us. I do intend to work on that, but who would have thought that pixel art animation takes forever... As you can see below, we’re far from done or looking good but it’s a start!

the real thing: REA

I started working on this before the GitHub repo REA came out. For those who don’t know, REA is a repo that can reverse engineer anything. And this means that SSB can be reverse engineered, and by the time I am writing this I think it already has been (yes it has been done here). This also means that I could actually use the recompiled PC version of Smash Ultimate as my RL training environment! One of the next steps once I am bored of making this little game would be to apply it to a richer environment like the actual SSB game.