A Neural Network-based StarCraft Bot Plays against Humans

We have tested a new version of our StarCraft bot against some skilled StarCraft players. These games were exciting to watch, especially because our bot used a neural network for build order planning. Videos showing two of these matches are included.

A Simple StarCraft Bot

A few month ago, I decided to develop a StarCraft bot from scratch. Setting up a hierarchy of modules for each sub-task was fairly straightforward, but it seemed like months of work to actually implement each module. So instead, I just programmed the bare minimum; just enough to get the bot to play and beat the built-in AI. This should be fine for now since my goal is to replace each module with neural networks.

The first version of the bot was uploaded a month ago to the Student StarCraft AI Tournament (SSCAIT). It has managed to win 34 out of its first 100 games, despite having some critical bugs in several of the modules. The most serious of these have now been fixed and the bot will be updated soon.

Deep Learning to Imitate Build Order Planning

In our recent paper Learning Macromanagement in StarCraft from Replays Using Deep Learning, we demonstrated how a neural network can learn to predict the next build (unit, building, upgrade or tech) produced by a human expert, with an accuracy of 44.4% (and top-3 accuracy of 87.1%). Furthermore, the trained network was integrated into UAlbertaBot, which managed to beat the built-in Terran AI in 68 out of 100 games.

net.png
Neural Network Architecture. The input layer consists of a state vector containing normalized values representing the number of each unit, building, technology, and upgrade in the game known to the player. Only a small subset is shown in the diagram for clarity. Three input values also describe the player’s supplies. The output of the network is the prediction of each build being produced next in the given state.

We have also extended our bot to use this network in games against the Terran race. All Protoss units are supported except for Dark Archons and Shuttles, which are removed from the network’s output.

build-order
The learned opening build-order when playing against the built-in Terran AI. The number next to each build icon represents the network’s probability of producing that build next. Points on the timescale indicate when the bot queried the network. In this example, the network always picked the build with the highest probability, while we usually sample an action using the probabilities in the output.

Playing against Humans

Several skilled StarCraft II players from Denmark had accepted to play against my bot. They seemed very experienced, but the matches were not entirely fair (to them) for several reasons:

  • They were playing against my bot in StarCraft: BroodWar, not StarCraft II.
  • They played on Astral Balance, a map that is never used in modern StarCraft matches.
  • They had to play in window mode (with a low screen resolution).
  • They had to use my predefined mouse settings.

Nevertheless, the games turned out to be very interesting, at least from my point of view. The humans did win, but they were definitely challenged in some parts of the game. For me, it was also surprising how well the bot actually played, considering that the neural network was trained on just 789,571 state-action pairs from 2,005 replays (actually only 80% of these were in the training set).

Here are two videos, showing the most exciting matches with our bot playing against human players:

You can speed up the video further in the YouTube settings.

Strengths

  • Somewhat adaptive: In some games, the strategy is adapting to new information. E.g. it sometimes goes for reavers or high templars versus biological units, or arbiters with stasis field against tanks.
  • Unpredictable: The build order varies enough to make the bot unpredictable. It can transition into zealots with speed, carriers, arbiters, reavers etc. This is probably due to 1) the probabilistic sampling of actions, 2) its occasional ability to adapt, and 3) its attempts to generalize to unseen situations, which are sometimes successful and sometimes not.
  • Expansion timings: It seems to expand at reasonable moments, usually when it has a solid army that can defend or push.

Weaknesses

  • Upgrades and research: The network has low probabilities for upgrade and research actions since these actions are sparse in the data set.
  • Missing information in the state: The neural network is not aware of resources on the map (available geysers, mined out minerals etc.), the terrain and unit positions. The network thus makes its predictions without knowing whether the bot is under attack.

What is Next

We have demonstrated that build order planning can be learned from replays, and we believe that this work can be improved by:

  • Training on a larger dataset such as STARDATA.
  • Use convolutional layers and include spatial information such as the mini-map.
  • Improve the network with reinforcement learning.

2 thoughts on “A Neural Network-based StarCraft Bot Plays against Humans

Add yours

  1. Why didn’t produce any units at moment it was over 3k minerals game 2. Looks like moving army left right without reason, don’t know what to do. When is the next big update? How much time until it reaches top bots level and capable of beating C/B rank icup level?

    Like

    1. I decided not to add minerals and gas as input to the network. Sometimes it decides to produce a gas-expensive unit/upgrade and it has to wait until it can afford it. After watching these games, I am will definitely add minerals and gas info to the input.

      The micromanagement and handling of armies in programmed by me and includes very simple (and probably stupid) rules. Only build order planning is done by the neural network.

      I will update the bot whenever I have some good results in our current projects. It will probably not reach C/B level without proper micromanagement, which I am not looking into, but other researchers are.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: