How to play text-based games with Microsoft’s TextWorld, a tool for training reinforcement learning agents

日本語の記事はこちらにあります。

Introduction

Microsoft released an open source project "TextWorld" last month.

TextWorld: A learning environment for training reinforcement learning agents, inspired by text-based games - Microsoft Research

TextWorld is a framework for training reinforcement learning agents using a text adventure game as a learning environment. It provides a function to automatically generate a text adventure game to let the agent play, and a programming interface to connect the game and agent. You can also play games on behalf of agents through the interface.

Text adventure game is the simplest form of an adventure game. All the interaction in the game is done with text, such as the depiction of a story or the choice of a player's actions. The origin of the name "adventure game" is the world's first text adventure game "Colossal Cave Adventure" released in 1976.

In this article, you can find the procedure for installing TextWorld on Amazon SageMaker (hereinafter called "SageMaker"), generating games automatically and playing the games by you.

Preparation

TextWorld requires Python 3 for running it. It supports only Linux and Mac at the moment. This post uses SageMaker (hereinafter called "SageMaker") for running TextWorld. Although the purpose of this post is to show how to play a game on TextWorld by a human, SageMaker environment will be useful for the future development of a reinforcement learning agent.

First, log in to the AWS management console then open the SageMaker dashboard. Select "Create notebook instance".

Enter the notebook instance name and select "Create new role" from the IAM role drop down list.

Check "None" for "S3 buckets you specify" then click "Create role".

Click "Create notebook instance".

Click "Open" to open Jupyter Notebook when the "Status" of the created notebook instance changes from "Pending" to "InService".

Select "Terminal" from the "New" drop-down list on the right side after the Jupyter Notebook opened. The following procedures are done in the terminal.

Install TextWorld

Clone the Git repository of TextWorld from GitHub then change directory to the created one.

$ git clone https://github.com/Microsoft/TextWorld.git
Cloning into 'TextWorld'...
(Omitted)
Resolving deltas: 100% (134/134), done.
$ cd TextWorld/

Install the package necessary for installing TextWorld with yum.

$ sudo yum install libuuid-devel libffi-devel -y
(Omitted)
Installed:
  libffi-devel.x86_64 0:3.0.13-16.5.amzn1                     libuuid-devel.x86_64 0:2.23.2-33.28.amzn1

Complete!

There are also packages that need to be installed via pip. Upgrade pip first to avoid a warning due to its version is old. Install the packages after upgrading pip.

$ pip install --upgrade pip
(Omitted)
Successfully installed pip-18.0
$ pip install -r requirements.txt

Install TextWorld.

$ pip install .
(Omitted)
Successfully built textworld
Installing collected packages: textworld
Successfully installed textworld-0.0.3

Four commands beginning with the prefix tw - are installed. Use tw-make for automatic generation of games andtw-play for playing games.

tw-data   tw-make   tw-play   tw-stats

Before running tw-make, you need to install pybars from pip to avoid ModuleNotFoundError.

$ pip install pybars3
(Omitted)
Installing collected packages: PyMeta3, pybars3
Successfully installed PyMeta3-0.5.1 pybars3-0.9.3

One more thing, Apply the following patch to the argparse included in Python 3 standard library to avoid another error. This error does not occur in Python 3.7 or later. The default Python version of the notebook instance was 3.6 at the time of writing.

--- /home/ec2-user/anaconda3/lib/python3.6/argparse.py  2018-04-29 16:18:42.000000000 +0000
+++ argparse.py 2018-08-13 02:30:36.317112960 +0000
@@ -1064,6 +1064,7 @@
                  prog,
                  parser_class,
                  dest=SUPPRESS,
+                 required=False,
                  help=None,
                  metavar=None):

@@ -1077,6 +1078,7 @@
             dest=dest,
             nargs=PARSER,
             choices=self._name_parser_map,
+            required=required,
             help=help,
             metavar=metavar)

$ patch /home/ec2-user/anaconda3/lib/python3.6/argparse.py argparse.patch
patching file /home/ec2-user/anaconda3/lib/python3.6/argparse.py

Generate a game

Let's generate a game to play with. Run tw-make command with the subcommand custom and options specified as follows.

$ tw-make custom --world-size 5 --nb-objects 10 --quest-length 5 --output gen_games/
Global seed: 25817
Game generated: gen_games/game_25817.ulx

The game is generated as a file in the format GLulx(.ulx file). --world-size specifies the number of rooms that make up the stage of the game, --nb-objects specified the number of objects appearing in the game, and --quest-length specified the minimum number of steps required to meet the goal of the game. The content of the game changes at random every time you generate it. which is displayed after generating the game. You can reconstruct the game with the same content if you specify the seed value with the option --seed.

Play with games

Play with the generated game

Let's play with the generated game. Run tw-play command with the location of the game you generated. A goal of the game and a description of the room at the start point are displayed then a prompt appears to wait for command input by you. Enter every action you want to do as a text. Achieve the goal by exploring rooms and by manipulating objects in the game.

$ tw-play gen_games/game_25817.ulx
Using GitGlulxMLEnvironment.




It's time to explore the amazing world of TextWorld! Here is your task for
today. First step, retrieve the passkey that's in the studio. Then, unlock the
door in the studio. Then, ensure that the door in the studio is open. After
that, make an attempt to go to the north. Then, retrieve the Advent Calendar
from the toolbox. Got that? Good!

-= Studio =-
You are in a studio. A standard kind of place.

You make out a stand. The stand is ordinary. But oh no! there's nothing on this
piece of garbage. What, you think everything in TextWorld should have stuff on
it?

There is a closed door leading north.

There is a passkey on the floor.

>

You can see a list of acceptable commands by entering help.

> help
Available commands:
  look:                                describe the current room
  goal:                                print the goal of this game
  inventory:                           print player's inventory
  go <dir>:                            move the player north, east, south or
west
  examine <something>:                 examine something more closely
  eat <something>:                     eat something edible
  open <something>:                    open a door or a container
  close <something>:                   close a door or a container
  drop <something>:                    drop an object on the floor
  take <something>:                    take an object that is on the floor
  put <something> on <something>:      place an object on a supporter
  take <something> from <something>:   take an object from a container or a
supporter
  insert <something> into <something>: place an object into a container
  lock <something> with <something>:   lock a door or a container with a key
  unlock <something> with <something>: unlock a door or a container with a key

You can always check the goal of the game with the command goal at any time. The goal of the game generated this time is to get a passkey at the studio and move north after opening the studio door and get the advent calendar in the toolbox.

> goal
It's time to explore the amazing world of TextWorld! Here is your task for
today. First step, retrieve the passkey that's in the studio. Then, unlock the
door in the studio. Then, ensure that the door in the studio is open. After
that, make an attempt to go to the north. Then, retrieve the Advent Calendar
from the toolbox. Got that? Good!

You can check the status of your location at any time with the look command. The room at the starting point is a studio, the passkey specified by the goal of the game is on the floor.

> look
-= Studio =-
You are in a studio. A standard kind of place.

You make out a stand. The stand is ordinary. But oh no! there's nothing on this
piece of garbage. What, you think everything in TextWorld should have stuff on
it?

There is a closed door leading north.

There is a passkey on the floor.

Pick up the passkey on the floor.

> take the passkey
Taken.

You can check what you got with the command inventory.

> inventory
You are carrying:
  a passkey
  a berry
  a key
  a book
  a pen

It seems there were some things at the beginning of the game. Try to eat a berry.

> eat my berry
You eat the berry. Not bad.

The berry has disappeared from your belongings as expected.

> inventory
You are carrying:
  a passkey
  a key
  a book
  a pen

Let's get back to the goal. Unlock the studio door with the passkey you got.

> unlock the door with the passkey
You unlock door.

Now you can go north. You need to explicitly "open" the door before going north.

> go north
You have to open the door first.

> open the door
You open door.

Now you are in a workshop after going north.

> go north

-= Workshop =-
Well I'll be, you are in a place we're calling a workshop. You decide to just
list off a complete list of everything you see in the room, because hey, why
not?

You can make out a toolbox. The toolbox contains an Advent Calendar. You can
make out a closed locker here.

There is a closed gateway leading east. There is an open door leading south. You
don't like doors? Why not try going west, that entranceway is unblocked.

There is a toolbox with an advent calendar specified by the goal in the room. Take the calendar.

> take the Advent Calendar from the toolbox
Taken.



                               *** The End ***

You scored 1 out of a possible 1, in 17 turn(s).


Would you like to RESTART, RESTORE a saved game, QUIT or UNDO the last command?
>

Done after 19 steps. Score 1/1.
$

The goal has been achieved, so the game is over. The number of steps you spent and The score obtained by the end of the game is displayed.

Play with existing games

The tw-play command also supports files in the format Z-machine. Since the Colossal Cave Adventure mentioned in "Introduction" can be found on the Internet in this format, Let's try the beginning part a little.

$ wget http://www.ifarchive.org/if-archive/games/zcode/Advent.z5
$ tw-play Advent.z5
Using FrotzEnvironment.

  ADVENTURE
  The Interactive Original
  By Will Crowther (1976) and Don Woods (1977)
  Reconstructed in three steps by:
  Donald Ekman, David M. Baggett (1993) and Graham Nelson (1994)
  [In memoriam Stephen Bishop (1820?-1857): GN]

  Release 9 / Serial number 060321 / Inform v6.31 Library 6/11 S

  At End Of Road
  You are standing at the end of a road before a small brick building. Around
  you is a forest. A small stream flows out of the building and down a gully.

>
> enter the building
   Inside Building                                Score: 36    Moves: 1
.
  Inside Building
  You are inside a building, a well house for a large spring.

  There are some keys on the ground here.

  There is tasty food here.

  There is a shiny brass lamp nearby.

  There is an empty bottle here.

>
> eat the food
   Inside Building                                Score: 36    Moves: 2
.
  (first taking the tasty food)
  Delicious!

>
>

Conclusion

How to install TextWorld on Amazon SageMaker and the procedure to automatically generate text-based games and play by humans introduced.

It is very interesting that computer games of the era when only text could be dealt with were attracting attention as a target of cutting edge machine learning research. I'd like to connect TextWorld with voice assistants such as Alexa to play TextWorld game via voice.

Thank you for reading until the end. I would be happy if you could share this article with SNS. I'm waiting for comments too!

Reference

• TextWorld: A learning environment for training reinforcement learning agents, inspired by text-based games - Microsoft Research
• GitHub - Microsoft/TextWorld: TextWorld is a sandbox learning environment for the training and evaluation of reinforcement learning (RL) agents on text-based games.
• Colossal Cave Adventure
• Machine Learning Models & Algorithms | Amazon SageMaker on AWS