How to play text-based games with Microsoft’s TextWorld, a tool for training reinforcement learning agents
日本語の記事はこちらにあります。
Introduction
Microsoft released an open source project "TextWorld" last month.
TextWorld is a framework for training reinforcement learning agents using a text adventure game as a learning environment. It provides a function to automatically generate a text adventure game to let the agent play, and a programming interface to connect the game and agent. You can also play games on behalf of agents through the interface.
Text adventure game is the simplest form of an adventure game. All the interaction in the game is done with text, such as the depiction of a story or the choice of a player's actions. The origin of the name "adventure game" is the world's first text adventure game "Colossal Cave Adventure" released in 1976.
In this article, you can find the procedure for installing TextWorld on Amazon SageMaker (hereinafter called "SageMaker"), generating games automatically and playing the games by you.
Preparation
TextWorld requires Python 3 for running it. It supports only Linux and Mac at the moment. This post uses SageMaker (hereinafter called "SageMaker") for running TextWorld. Although the purpose of this post is to show how to play a game on TextWorld by a human, SageMaker environment will be useful for the future development of a reinforcement learning agent.
First, log in to the AWS management console then open the SageMaker dashboard. Select "Create notebook instance".
Enter the notebook instance name and select "Create new role" from the IAM role drop down list.
Check "None" for "S3 buckets you specify" then click "Create role".
Click "Create notebook instance".
Click "Open" to open Jupyter Notebook when the "Status" of the created notebook instance changes from "Pending" to "InService".
Select "Terminal" from the "New" drop-down list on the right side after the Jupyter Notebook opened. The following procedures are done in the terminal.
Install TextWorld
Clone the Git repository of TextWorld from GitHub then change directory to the created one.
$ git clone https://github.com/Microsoft/TextWorld.git Cloning into 'TextWorld'... (Omitted) Resolving deltas: 100% (134/134), done. $ cd TextWorld/
Install the package necessary for installing TextWorld with yum.
$ sudo yum install libuuid-devel libffi-devel -y (Omitted) Installed: libffi-devel.x86_64 0:3.0.13-16.5.amzn1 libuuid-devel.x86_64 0:2.23.2-33.28.amzn1 Complete!
There are also packages that need to be installed via pip. Upgrade pip first to avoid a warning due to its version is old. Install the packages after upgrading pip.
$ pip install --upgrade pip (Omitted) Successfully installed pip-18.0 $ pip install -r requirements.txt
Install TextWorld.
$ pip install . (Omitted) Successfully built textworld Installing collected packages: textworld Successfully installed textworld-0.0.3
Four commands beginning with the prefix tw - are installed. Use tw-make for automatic generation of games andtw-play for playing games.
tw-data tw-make tw-play tw-stats
Before running tw-make, you need to install pybars from pip to avoid ModuleNotFoundError.
$ pip install pybars3 (Omitted) Installing collected packages: PyMeta3, pybars3 Successfully installed PyMeta3-0.5.1 pybars3-0.9.3
One more thing, Apply the following patch to the argparse included in Python 3 standard library to avoid another error. This error does not occur in Python 3.7 or later. The default Python version of the notebook instance was 3.6 at the time of writing.
--- /home/ec2-user/anaconda3/lib/python3.6/argparse.py 2018-04-29 16:18:42.000000000 +0000
+++ argparse.py 2018-08-13 02:30:36.317112960 +0000
@@ -1064,6 +1064,7 @@
prog,
parser_class,
dest=SUPPRESS,
+ required=False,
help=None,
metavar=None):
@@ -1077,6 +1078,7 @@
dest=dest,
nargs=PARSER,
choices=self._name_parser_map,
+ required=required,
help=help,
metavar=metavar)
$ patch /home/ec2-user/anaconda3/lib/python3.6/argparse.py argparse.patch patching file /home/ec2-user/anaconda3/lib/python3.6/argparse.py
Generate a game
Let's generate a game to play with. Run tw-make command with the subcommand custom and options specified as follows.
$ tw-make custom --world-size 5 --nb-objects 10 --quest-length 5 --output gen_games/ Global seed: 25817 Game generated: gen_games/game_25817.ulx
The game is generated as a file in the format GLulx(.ulx file). --world-size specifies the number of rooms that make up the stage of the game, --nb-objects specified the number of objects appearing in the game, and --quest-length specified the minimum number of steps required to meet the goal of the game. The content of the game changes at random every time you generate it. which is displayed after generating the game. You can reconstruct the game with the same content if you specify the seed value with the option --seed.
Play with games
Play with the generated game
Let's play with the generated game. Run tw-play command with the location of the game you generated. A goal of the game and a description of the room at the start point are displayed then a prompt appears to wait for command input by you. Enter every action you want to do as a text. Achieve the goal by exploring rooms and by manipulating objects in the game.
$ tw-play gen_games/game_25817.ulx Using GitGlulxMLEnvironment. It's time to explore the amazing world of TextWorld! Here is your task for today. First step, retrieve the passkey that's in the studio. Then, unlock the door in the studio. Then, ensure that the door in the studio is open. After that, make an attempt to go to the north. Then, retrieve the Advent Calendar from the toolbox. Got that? Good! -= Studio =- You are in a studio. A standard kind of place. You make out a stand. The stand is ordinary. But oh no! there's nothing on this piece of garbage. What, you think everything in TextWorld should have stuff on it? There is a closed door leading north. There is a passkey on the floor. >
You can see a list of acceptable commands by entering help.
> help Available commands: look: describe the current room goal: print the goal of this game inventory: print player's inventory go <dir>: move the player north, east, south or west examine <something>: examine something more closely eat <something>: eat something edible open <something>: open a door or a container close <something>: close a door or a container drop <something>: drop an object on the floor take <something>: take an object that is on the floor put <something> on <something>: place an object on a supporter take <something> from <something>: take an object from a container or a supporter insert <something> into <something>: place an object into a container lock <something> with <something>: lock a door or a container with a key unlock <something> with <something>: unlock a door or a container with a key
You can always check the goal of the game with the command goal at any time.
The goal of the game generated this time is to get a passkey at the studio and move north after opening the studio door and get the advent calendar in the toolbox.
> goal It's time to explore the amazing world of TextWorld! Here is your task for today. First step, retrieve the passkey that's in the studio. Then, unlock the door in the studio. Then, ensure that the door in the studio is open. After that, make an attempt to go to the north. Then, retrieve the Advent Calendar from the toolbox. Got that? Good!
You can check the status of your location at any time with the look command.
The room at the starting point is a studio, the passkey specified by the goal of the game is on the floor.
> look -= Studio =- You are in a studio. A standard kind of place. You make out a stand. The stand is ordinary. But oh no! there's nothing on this piece of garbage. What, you think everything in TextWorld should have stuff on it? There is a closed door leading north. There is a passkey on the floor.
Pick up the passkey on the floor.
> take the passkey Taken.
You can check what you got with the command inventory.
> inventory You are carrying: a passkey a berry a key a book a pen
It seems there were some things at the beginning of the game. Try to eat a berry.
> eat my berry You eat the berry. Not bad.
The berry has disappeared from your belongings as expected.
> inventory You are carrying: a passkey a key a book a pen
Let's get back to the goal. Unlock the studio door with the passkey you got.
> unlock the door with the passkey You unlock door.
Now you can go north. You need to explicitly "open" the door before going north.
> go north You have to open the door first. > open the door You open door.
Now you are in a workshop after going north.
> go north -= Workshop =- Well I'll be, you are in a place we're calling a workshop. You decide to just list off a complete list of everything you see in the room, because hey, why not? You can make out a toolbox. The toolbox contains an Advent Calendar. You can make out a closed locker here. There is a closed gateway leading east. There is an open door leading south. You don't like doors? Why not try going west, that entranceway is unblocked.
There is a toolbox with an advent calendar specified by the goal in the room. Take the calendar.
> take the Advent Calendar from the toolbox
Taken.
*** The End ***
You scored 1 out of a possible 1, in 17 turn(s).
Would you like to RESTART, RESTORE a saved game, QUIT or UNDO the last command?
>
Done after 19 steps. Score 1/1.
$
The goal has been achieved, so the game is over. The number of steps you spent and The score obtained by the end of the game is displayed.
Play with existing games
The tw-play command also supports files in the format Z-machine. Since the Colossal Cave Adventure mentioned in "Introduction" can be found on the Internet in this format, Let's try the beginning part a little.
$ wget http://www.ifarchive.org/if-archive/games/zcode/Advent.z5 $ tw-play Advent.z5 Using FrotzEnvironment. ADVENTURE The Interactive Original By Will Crowther (1976) and Don Woods (1977) Reconstructed in three steps by: Donald Ekman, David M. Baggett (1993) and Graham Nelson (1994) [In memoriam Stephen Bishop (1820?-1857): GN] Release 9 / Serial number 060321 / Inform v6.31 Library 6/11 S At End Of Road You are standing at the end of a road before a small brick building. Around you is a forest. A small stream flows out of the building and down a gully. > > enter the building Inside Building Score: 36 Moves: 1 . Inside Building You are inside a building, a well house for a large spring. There are some keys on the ground here. There is tasty food here. There is a shiny brass lamp nearby. There is an empty bottle here. > > eat the food Inside Building Score: 36 Moves: 2 . (first taking the tasty food) Delicious! > >
Conclusion
How to install TextWorld on Amazon SageMaker and the procedure to automatically generate text-based games and play by humans introduced.
It is very interesting that computer games of the era when only text could be dealt with were attracting attention as a target of cutting edge machine learning research. I'd like to connect TextWorld with voice assistants such as Alexa to play TextWorld game via voice.
Thank you for reading until the end. I would be happy if you could share this article with SNS. I'm waiting for comments too!
Reference
- TextWorld: A learning environment for training reinforcement learning agents, inspired by text-based games - Microsoft Research
- GitHub - Microsoft/TextWorld: TextWorld is a sandbox learning environment for the training and evaluation of reinforcement learning (RL) agents on text-based games.
- Colossal Cave Adventure
- Machine Learning Models & Algorithms | Amazon SageMaker on AWS
