We have tested a new version of our StarCraft bot against some skilled StarCraft players. These games were exciting to watch, especially because our bot used a neural network for build order planning. Videos showing two of these matches are included.
A Simple StarCraft Bot
A few month ago, I decided to develop a StarCraft bot from scratch. Setting up a hierarchy of modules for each sub-task was fairly straightforward, but it seemed like months of work to actually implement each module. So instead, I just programmed the bare minimum; just enough to get the bot to play and beat the built-in AI. This should be fine for now since my goal is to replace each module with neural networks.
The first version of the bot was uploaded a month ago to the Student StarCraft AI Tournament (SSCAIT). It has managed to win 34 out of its first 100 games, despite having some critical bugs in several of the modules. The most serious of these have now been fixed and the bot will be updated soon.
Deep Learning to Imitate Build Order Planning
In our recent paper Learning Macromanagement in StarCraft from Replays Using Deep Learning, we demonstrated how a neural network can learn to predict the next build (unit, building, upgrade or tech) produced by a human expert, with an accuracy of 44.4% (and top-3 accuracy of 87.1%). Furthermore, the trained network was integrated into UAlbertaBot, which managed to beat the built-in Terran AI in 68 out of 100 games.
We have also extended our bot to use this network in games against the Terran race. All Protoss units are supported except for Dark Archons and Shuttles, which are removed from the network’s output.
Playing against Humans
Several skilled StarCraft II players from Denmark had accepted to play against my bot. They seemed very experienced, but the matches were not entirely fair (to them) for several reasons:
- They were playing against my bot in StarCraft: BroodWar, not StarCraft II.
- They played on Astral Balance, a map that is never used in modern StarCraft matches.
- They had to play in window mode (with a low screen resolution).
- They had to use my predefined mouse settings.
Nevertheless, the games turned out to be very interesting, at least from my point of view. The humans did win, but they were definitely challenged in some parts of the game. For me, it was also surprising how well the bot actually played, considering that the neural network was trained on just 789,571 state-action pairs from 2,005 replays (actually only 80% of these were in the training set).
Here are two videos, showing the most exciting matches with our bot playing against human players:
You can speed up the video further in the YouTube settings.
- Somewhat adaptive: In some games, the strategy is adapting to new information. E.g. it sometimes goes for reavers or high templars versus biological units, or arbiters with stasis field against tanks.
- Unpredictable: The build order varies enough to make the bot unpredictable. It can transition into zealots with speed, carriers, arbiters, reavers etc. This is probably due to 1) the probabilistic sampling of actions, 2) its occasional ability to adapt, and 3) its attempts to generalize to unseen situations, which are sometimes successful and sometimes not.
- Expansion timings: It seems to expand at reasonable moments, usually when it has a solid army that can defend or push.
- Upgrades and research: The network has low probabilities for upgrade and research actions since these actions are sparse in the data set.
- Missing information in the state: The neural network is not aware of resources on the map (available geysers, mined out minerals etc.), the terrain and unit positions. The network thus makes its predictions without knowing whether the bot is under attack.
What is Next
We have demonstrated that build order planning can be learned from replays, and we believe that this work can be improved by:
- Training on a larger dataset such as STARDATA.
- Use convolutional layers and include spatial information such as the mini-map.
- Improve the network with reinforcement learning.