Training your first AI with Unity ML-Agents

ML-Agents is an open-source project that enables games and simulations to serve as environments for training intelligent agents.

Introduction

You’re familiar with Unity and you like machine learning ? Well, this toolkit is made for you ! Unity’s ML-Agents solution gives you a way to quickly and efficiently develop and test new AI algorithms for a new generation of games. With Unity ML-Agents, a variety of training scenarios are possible, depending on how agents, brains, and rewards are connected.

I - Installation of ML-Agents

I assume that you already have Unity. First, start by creating a new empty 3D project and open up the package manager window. Make sure that preview packages is enabled. Search for ML-Agents and install it.

In order to start, it is better to use the basic examples that are given by Unity themselves. You need to download their repository.

Decompress the downloaded repository, open the Projects/Assets folder and drag it to Unity editor. You will have few examples. Let’s say that the one interesting us is 3D ball. If you hit play, you’ll see that the project is already working. Unity gave us the training model within the project. This little blue head are balancing the ball quite perfectly. The behavior is learned, not hardcoded : it proves that this toolkit uses well machine learning.

II - Vocabulary to understand

How ML-Agents works

The two important scripts in the 3D ball are the behavior and the ball 3d agent script.

A - Agent

Unity Component. Agent is an actor. An agent execute action, collection observations and assign rewards. Agents actions are determined by decisions produced by a Policy.

B - Behavior

Multiple agent can be liked to multiple behavior. A behavior determine how agents act. It recieves observations and rewards and send back actions.

C - Observation

Partial information describing the state of the environment available to a given agent. (e.g. Vector, Visual)

D - Vector

Vector observations

Vector observations is a float array of a space x (imagine space = 8, the float array can fit 8 elements) that can be stacked. We can stack these observations to wait before sending. It gather a new vector observation every time a decision is requested and it will simply add the new vector to the running stacked history, while bumping the oldest observation out of the stacked vector array.

Vector actions

The carrying-out of a decision on the part of an agent within the environment. It has two types : continuous and discrete. Continuous is a float with ranging from -1 to 1. Discrete is an integer. You choose the type depending on the game you’re coding. Discrete numbers are used to encode different decisions like direction while continuous numbers are for more complexe decisions like the trajectory.

E - Model and behavior

Model is the .nn file that is the trained model. At first, the model is already here because unity already have trained their model. You’ll be able to created yours later ! If you have some GPUs you can change Inference Device from CPU to GPU in the behavior parameters.

There is 3 types of behavior :

• Heuristic : the classical way AI works, which is hardcoding the action

• Default : learning : whichi is when the AI is training with machine learning and generating models at the same time

• Inference : which is when the model has been created and that your AI doesn’t need to learn anymore : you juste use the trained model.

F - Academy

Globally accessible singleton object which controls timing, reset, and training/inference settings of the environment (in previous version it was attached to a GameObject). The Academy controls the training process for all agents with external Python process and ensures that all agents are in synch.

III - Method from the Agent script

If you create your own agent, you need to make it inherit from the Agent class. Now, let’s get through the agent class a little bit with these important method.

Initialize

Initialize is called when the game object gets enabled : it is between the await and start method. It allows the agent to connect to its behavior, possible sensors and academy.

CollectObservations

Everything your AI needs to make decision is collected by this method.

OnActionReceived

Where you transform action parameter into concrete actions : whatever your AI is doing : moving, jumping, turning…

OnEpisodeBegin

It last until the objective has either been achieved or failed (it resets in this case). In the 3D ball example, it lasts until the ball has fallen.

IV - Installation of Python and mlagents

You first need to download Python. It is better to use the 3.6 or 3.7 version. Make sure you install pip as well.

Then you need to install mlagents by typing the command : pip3 install mlagents

V - Training your first AI

Congrats ! You have Python and mlagents installed ! The next you need to do is running your console where you have downloaded the github project of Unity. Make sure that you are at the root of it.

Then, type : mlagents-learn config/trainer_config.yaml –run-id=FirstAI

You’ll see your terminal running and then the logo of Unity. You’ll need to hit the play button into the Unity engine. Now, it’s time to wait a little bit ! Your model is training.

When it is done, you can copy it into the Project/Assets/Example/3DBall/TFModels and choosing it as NN model.

Conclusion

You can now start to make your own AI system game thanks to Unity Machine Learning Agents.

📖 Sources

Websites

Name	Author	Date	Description
Unity3D Machine learning	Unity	2017	All contents about Machine learning with Unity
towardsdatascience	Thomas Simonini	2020	An Introduction to Unity ML-Agents
Unity Machine Learning Agents	Unity	2017	Machine Learning Agents