Last revision May 2022

Today I will try to explain how to create a program that will play your favorite video games. This is useful when you don't have time to farm and you need to kill moobs and get items.

If you are interested in joining to this project, you are welcome to contribute.

The RPG game that I will choose for this project is Genshin Impact

but also this program should be compatible with New World , World of Warcraft, Guild Wars 2 ,Final Fantasy XIV, Elder Scrolls Online and so on.

Currently due to lack of time, just I will develop this interesting project on Genshin Impact, that can be used as a template to use to the remaining games.

The objective of the project is develop a

a) general neural network that can be adapted to different videogames that can be useful to automatize video games.

b) develop the local/cloud infrastructure that allows us record and process the data.

The cloud technologies that can be used in principle can be AWS, Google Cloud and Microsoft Azure. We can try to test each of them.

We will start with local development and some free AWS microservices.

c) Enjoy coding and pass fun, one of the important aspects of this project is to learn, enjoy and pass good moments with the AI Community Gamer.

The first tasks that we want to perform are:

Create a Neural Network that allow the following:

1. Auto exploration
2. Kill moobs in a Dungeon
3. Collect items.

The first challenge that we want to perform is develop an Neural Network that allow us run from the Initial Point A until final Point B

we will consider the initial point Mondstand and the Final point Thousand Wind Temple

The time that we will choose is 12:00

There are two different versions of the same program.

Type 1- The type blog, via Jupyter Notebook in the format .ipynb

Type 2- The type prod, via Python by using command Prompt.

The Type 1, the non prod, is the self-explanatory program written in Jupyter notebook.

The reason to do that is familiarize with the code, and see the logic the development. In this way anyone who is interested to understand why I have made the program in this way.

This help to newcomers introduce the code , and help find bugs. In addition allows write the code type blog.

The Type 2 is the production program, written in simply python script, without discussions or extra things that are not needed to run the whole program. Here we skip all the analysis of the type 1 and just we put everything a clean as a possible.

The preliminary solution is given with the following parts:

1. Creation of the frontend application - Here, we create a python program that will record our gameplay and control our videogame. The frontend application consists into 3 programs.

   a) Collection of Data - This program record the screen during the gameplay. At this stage we focus with small amount of data. Later in the Part 2 we will allocate the data in the cloud.

   b) Creation of the Model - This program read the collected data and we create a neural network model

   c) Test the model - Here we load the created model and test in the gameplay.

   (updated March 2022).

   There is one program to record the gameplay and there is another that simulates the gameplay

   Gameplay API programs

   

2. Creation of the backend pipeline . The extension of the frontend part consists to create a pipeline program that will move the recorded data into the cloud. This is important to record log periods of gameplay recording. Here we can use object storage like s3 or minio. This allows us do not to saturate the hard disk on the client side during the gameplay.

3. Creation of the Dataset - In this part, we load the a dataset of the gameplay, that contains the screen images, the recorded keys and gamepad.

4. Data wrangling -We can improve our dataset by performing the augmentation procedure of the images. Here we need to apply good skills in computer vision.

5. Model creation - The first version of this bot will use Neural Networks. The framework used will be Tensorflow and Pytorch. Here we will create the brain of our bot.

6. Training g Docker Container - Creation o the Docker container environment with JypyterLab for this Bot project with all the training packages.

7. Training of the model - Here, we need to train the neural network. Here we may require a cluster. We can use the Colab to use the training with colab or pay a little to run a cluster on EMR AWS with a Container with all our environment ready.

8. Creation of the frontend application that plays - Here, we need to create a program that reads the game's screen and depends on what executes the Artificial intelligence Model you have made before.

For the development of the Neural Network we have two approaches:

The first approach is consider the preprocessed images, that means take the rgb images and the input into an array (450, 480, 270, 3) without processing the data and then use an inception_v3 model with 29 ouputs.

The second experimental approach consists to take the images and filter them into unicolor, adapted to the path where the characters run, then append this unicolor image with the input into an array (450, 480, 270, 1) and then use an inception_v3 model with 29 ouputs. The motivation to use unicolor is just a manner to simplify the image such is explained in this blog

Currently, the stable version of the BOT-MMORPG is the pre-processed version.

Pre-processed version

In this version, you will need to run first

1-Collect_Data_Preprocessed-rgb.ipynb or 1-collect_data.py

When creating training data, this works when you have the game, in fullscreen mode, 1980x1080 resolution.

You need this for both training and testing.

Next, Train the model .

With the preprocessed data we will consider the inception_v3 model.

2-train_model_preprocessed.ipynb or 2-train_model.py

Finally, we test our model in game with

3-test_model.ipynb or3-test_model.py

Processed version

The processed version is the upcoming version where we are trying to clean more our data and improve the neural network.

2-Cleaning-Data.ipynb

2-train_model_cleaned.ipynb

3-test_model-processed.ipynb

(There are some experimental tests use OpenCV to detect the ways to walk here , inspired to use U-Net models.

and also we try unicolor versions )

Things to do before execute our bot-mmorpg:

1. Open your Genshin Impact Game
2. Go infront of the bridge of Mondstat
3. Change the time to 12:00
4. Verify that you are using your a controller.
5. Run the main()
6. Return to the windows of your game

To abort the code, go to your command prompt terminal where your notebook where opened, and there you can press ESC to abort the program or T pause the AI program.

Currently we are using the Inception_v3 model, but we are interested to perform transform learning from the following models:

And identify the behavior with our train and test dataset.

• Everyone at the gamePyd proyect
• Everyone at vJoy for the vXboxInterface DLL
• nefarius for ScpVBus
• See also the list of contributors who participated in PYXinput
• This project it is possible thank you the previous projects Self-Driving-Car-with-Neural-Networks
• Everyone at PyGTA5 project

If you are already part of the team the official channel in slack is:

https://aws-ml-group.slack.com/

Please free to contribute following the standard guidelines for submitting patches and additions or solutions. Feel free to submit issues and enhancement requests.

To more information visit ruslanmv.com.

Copyright 2021 Ruslan Magana Vsevolodovna This program is distributed under the terms of the GNU Lesser General Public License.