1. Goal
   

2. Old School RuneScape & RuneScape (3)
   

3. Notable Achievements & Facts
   

4. Current Capabilities Demonstrations
   

   4.1 Environmental Interaction Using Object Detection
   

   4.2 Simulation Environment GUI Software
   

5. Autonomous Navigation Problem Potential Solutions & Key Topics
   

The development of an AI-based video game bot for the Massively Multiplayer Online Role-Playing Game (MMORPG) titled Old School RuneScape, the former redux version of RuneScape (AKA RuneScape 3), developed by the British video game studio, Jagex Limited.

Old School RuneScape, originally named RuneScape, is an improved version of the backed up source code of the version of the game from August 2007. It was brought back on the 22nd of February, 2013, after having been temporally taken down so as to make way for RuneScaoe 3, the third and most recent iteration of the game, launched in July in 2013. Despite this latest HTML5-client based version of the game (RuneScape 3) possessing graphical effects and features far more advanced compared to that of Old School RuneScape, it is this earlier version of the game (OSRS) which happens hosts the largest base of players between the two versions¹.

In-game screenshot comparison of both games. RuneScape3 (left), showing a player carrying a longsword and standing amongst a bunch of NPC wizards, a typical scenario for an experienced player. Old School RuneScape (right), showing a new player in combat with a goblin, a standard activity every newly joined player goes through.

• Guinness World Records award for the "Most users of an MMO videogame" with a reported number of over 254,994,744 player accounts having been created since the game was first launched back in 2001.

• Guinness World Records award for the "Most prolifically updated MMORPG videogame" reportedly having been updated once a week on average for a total number of updates greater than 1,014 since the games initial launch.

• Guinness World Records award for the "Greatest aggregate time playing an MMO or MMORPG videogame (all players)" for having a total aggregate number of minuets of player game time spent exceeding 443 billion, counting as of 27 July 2012.

• An anti-botting system, called "ClusterFlutterer", released in an update on the 25th of October, 2012, under the nickname of "Bot-Nuke", eventually

lead to the banning of an estimated 98% of all botting accounts, equating approximately to 7.7 million million accounts being banned².

Autonomous slaying of cows by the bot with the help of object detection. The TensorFlow pre-trained model which was used was ssd_mobilenet_v1_coco and it was trained on a dataset of images and PASCAL VOC format annotations, created using LabelImg.

Object detection based autonomous NPC slaying at the cow pen near the in-game city/town called Lumbridge

Object detection based autonomous resource gathering at the mines south of Lumbridge

An extensive still-in-development simulation GUI program developed for various purposes. Some of these are:

• Use of as a controllable & deterministic simulation environment.

• Implementation & testing of reinforcement learning algorithms.

• Assistive experimentation tool for autonomous navigation.

• Investigation of various planning & decision making algorithms.

• Validation & verification of implementations prior to integration with the system.

A great and invaluable piece of contribution to project by Victor Guillet.

An extensive still-in-development simulation GUI program developped for various purposes. Is currently mainly being used for implementing and testing reinforcement learning algorithms as streamlined and conveniently as possible, in addition to being used as a helpful tool in the quest of solving the problem of autonomous navigation. Credits to Victor Guillet for his invaluable contribution

Object detection based autonomous NPC slaying at the cow pen near the in-game city/town called Lumbridge

__## Autonomous Navigation Problem Potential Solutions & Key Topics

• Iterative Closest Point (ICP) algorithm

• End-to-end (E2E) Deep Learning (DL)

  • CNN2-LSTM Network

  • SuperPoint network

  • Hierarchical Scene Coordinate network (hscnet) for coordinate classification (& regression?) for visual localisation

  • Hierarchical Localisation network

• Extended Kalman Filter (EKF)

• Dead Reckoning

• Topological (visual) graphs & maps

• View graphs (& maps)

• Optical flow

• Key points

• Global & local feature descriptors

• Simultaneous Localisation And Mapping (SLAM)

  • Occupancy Grid SLAM

  • GraphSLAM

  • RGB-D SLAM

Flow of steps:

• Starting Place: Set starting place (global) Main Map loc coordinates

• Along Path/Trajectory:

  • 1st) Convert A* generated (global) Main Maps displacements to mini map (mm) screen pixel pos coords.

  • 2nd) Use generated mm screen pixel pos coords for Left Mouse Button actuator clicks.

• settings.py: This script is reponsible of setting all the necessary settings which will me required and used (in the background) by the various other scripts/modules present in this repo. The general settings in question are:
  • Directory and file paths.
  • The specific mouse & keyboard buttons to be monitored, and their initial states upon initialization.
  • The size/dim (width & height) to resize (& maintain) the game window, and the position (x, y)( w/ (0, 0) at the top left of the screen) to position the game window to. Also, the name of the window.

• jarvis/tests/osrs_api_test.py: This script is used for testing the Old School Runescape API which allows you to fetch real-time market prices of items from the Grand Exchange.
• jarvis/tests/test_game_client_wndw.py: This script is used for testing the game client handler. The primary purpose of the GameClient class is to "take control" of the game client window in order to resize & reposition the game client window.
• jarvis/tests/test_slam.py: Does not seem to be working.
• jarvis/tests/test_vision_sys.py:
• jarvis/tests/test_vision_sys_GUI_version.py: This script is for testing the integration of following modules:
  • Object Detection & Tracking: Vision
  • Game Client Handler, GameClient
  • GUI: VisionTestGUIHandler & VisionSysHelperUtil
  • Actuators (Mouse & Keyboard Clicks): Mouse & HardwareEventsListener
  • The actual agent/bot: Hobbes

• jarvis/utils/vision_sys_helper_util.py: The VisionSysHelperUtil class in this script is a helpful utility for the Vision & VisionTestGUIHandler modules. Its utilisation purpose is the annotation of infomartion and drawing of "symbols" on the captured screen shots which are displayed in the GUI. All "settings" of this class are True by default! 
  VisionTestGUIHandler class with (left) all default/True settings and (right) with every thing "switched" off. Note that the script in use (being annotated in the bottom right) is jarvis/tests/test_vision_sys_GUI_version.py script.

• jarvis/game_client/game_client.py: This class is reponsible of the managinging & handling of the OSRS game client window. For instance, it allows for the changing of "the position and dimensions of the specified window". Note that "For a top-level window, the position and dimensions are relative to the upper-left corner of the screen." Also, "NOTE (regarding window alignment with true left & true top) "Windows 10 has an invisible border of 7 pixels (Totaling to 8 pixels if you include the visible 1 pixel window border.)"

• jarvis/vision_sys/vision_cls.py:
• jarvis/vision_sys/sensor.py:
• jarvis/vision_sys/obj_detector.py:
• jarvis/vision_sys/obj_tracker.py:

• test_vision_sys_GUI_version.py
• vision_sys/percept_cls.py
• vision_sys/vis_toolkit.py
• vision_sys/vision_cls.py

• game_client/game_client.py
• utils/screen_tools.py
• tests/test_screen_tools.py

• navigation/*
• tests/test_slam.py

• utils/screen_tools.py
• tests/test_screen_tools.py

Script Documentation TO BE ADDED

• jarvis/tests/test_vision_sys_GUI_version.py
• jarvis/utils/vision_sys_helper_util.py
• jarvis/game_client/game_client.py

Files TO BE DELETED

• README_backup.adoc
• test_vision_sys_GUI_version.spec
• draft_for_README.md
• Handbook of Computer Vision Algorithms in Image Algebra.pdf

INCOMPLETE

Folders TO BE DELETED

• bot_sys_components
• build
• dist

INCOMPLETE

JARVIS Folders & Files In Use [X] & Not In Use [_]

• jarvis/actuator_sys/actuator.py
• jarvis/actuator_sys/mouse.py
• jarvis/game_client/game_client.py
• jarvis/hobbes_bot/basic_self_contained_bot_cls.py
• jarvis/hobbes_bot/hobbes.py
• jarvis/hobbes_bot/inventory.py
• jarvis/hobbes_bot/navigation.py
• jarvis/hobbes_bot/skills.py
• jarvis/jarvis_core/agent.py
• jarvis/jarvis_core/environment.py

INCOMPLETE

• "def": "default"
• "w/": "with"
• "w/o": "without"
• "b/c": "because"
• "args": "arguments"
• "params": "parameters"
• "OSRS": "Old School RuneScape"
• "RS3": "RuneScape 3"
• "vis" or "viz: "Vision"
• "h/w": "hardware"
• "pos": "position"
• "coords": "coordinates"
• "dim(s)": "dimension(s)"
• "obj": "object"
• "dect" (yes, dect): "detection"
• "wndw": "window"
• "num": "number"
• "req": "required"
• "V&V" or "VV": "Verification & Validation"
• "sys": "system"

INCOMPLETE

^*More accurately, Old School RuneScape Artificial Intelligence, OSRS-AI, Bot

¹Old School RuneScape Official Wiki Webpage

²Runescape bot nuking event bans 1.5 million bots in one day