November means ProcJam and NaNoGenMo, and while I neither joined the jam nor gen’d a no’, the festive mood inspired me to dive into Kyle McDonald’s fantastic roundup of his recent work in machine learning. That lead me to finally read some of Andrej Karpathy’s posts on neural networks, and particularly this detailed post on using recurrent neural networks to generate text. A particular treat of Karpathy’s work is that not only is his text RNN freely available, there’s even an updated/improved version by Justin Johnson, and there’s a detailed installation guide for OSX by Jeff Thompson. While I still did have a bit of installation trouble, I was able to get through it with some help from Trevor and a few Stack Overflow searches.
So, what to generate? Back in my time at Recurse Center (called Hacker School at the time), I did a quick implementation of a Markov chain text generator from the Inform 7 source code of Bronze. The results were hilarious, as is typical for Markov chain output, but very little was workable Inform 7 code. Since, as McDonald and Karpathy point out, RNNs are much more capable of capturing structure than Markov chains are, why not repeat the Inform generation project with an RNN instead?
I knew I’d need a larger corpus than Bronze, which is only 253 KB. Luckily, there is a fair amount of Inform 7 source code available. I initially trained the RNN with the full source text of Blue Lacuna, by Aaron Reed, because it is well known as a massive and intricate work. (Indeed, the source is 2.4 MB.) However, I didn’t love the results. Blue Lacuna’s source is dense, with not a lot of white space, and the output ran together interminably. Also, the consistent imagery and central non-player characters made for results that were obviously lifted from Blue Lacuna, such as name-dropping the characters repeatedly.
Which is not to say that some of the results weren’t poetic:
Ãd not vanimus of Progue is false;
if the player is a beat in DenialC with reaction "'Well and strange the surface of the strange stream the stars the surface of the stars the surface of the strange the stars the start of the surface of the start to the stars the start of the stars the start to the start to the stars the stars the stars the stars the stars the start and stretches the stars and stream stream beats and seems to the stars the stars the stars the stars and strange the stars and stream beats and the stars and stretches and sea an explain to the start of the start of the stars and strange the stars and sea and the stars and strange stream and stream stream and strange the stars the stars of the stream and strange the surface of the stars and strange stream beats and strange the stars the stars of the stars the stars and strange the surface of the compass and strange the stars to the stars and the start and the start to the stars and start to the stars the start to the surface of the stars of the stars the stars and sea and stream the stars and strange the stars the start of the surface of the stars the surface of the stars the stars of the start to the strange the stars the start of the stars the start to the start to the surface of the stars the stars the surface of the strange strange the stars the stars the stars the stars the surface of the stars and stream beats and strange the stars and start to the staries and stream and strange strange strange stream point of the stars the stars and strange the stars the stars and strange the stars to the stars to the start to the stars and strange the stars the start to the stars the stars and strange the stars the start to the stars the start of the stars and stream beats and shadows and strange stream to the stars the surface of the stars and strange the stars and strange stream stream beats and strange the beach of the stars and strange stream the stars the surface of the stars of the stars the stars the start and strange the story and start of the stars the start to the stars of the start to the surface of the stars the stars and the stars and strange the surface of the strange strange strange stream the stars the start to the start to the stream strange the stars and strange the surface of the stars the stars the surface of the start of the stars and stream to the stars the stars the stars and start to the stars and stretches and sea and seems to the stars and strange stretching the start of the surface of the stars the stars the start of the stars the stars
(This went on for all 2,000 requested characters. In the RNN’s defense, I had requested a low “temperature,” meaning fewer opportunities for the RNN to break out of its obsessive loop.)
I needed a bigger corpus, but I felt like I would get the best results if I still stuck to a single author with a reasonably consistent style. So I assembled a corpus out of all of Emily Short’s games that are listed with source on the IFDB. For good measure, I added the code from all the examples in the Inform 7 Recipe Book, since many of these were also written by her and they maintain a consistent formatting style. (To get just the code, I copied the last part of the plain text Recipe Book file, with all the examples, and deleted lines that didn’t start with at least one tab character. I then deleted the first tab character from all the remaining lines.) Together, this made a 3.4 MB document.
Training took about three hours without hardware acceleration, and I liked the results generated at temperatures around 0.4. Here are my favorite thousand or so characters from the first eight thousand I requested:
"Ben"
The Screening is a room.
Carry out looking rule:
if the noun is not a view of the player;
if the player is in the player;
say "[The noun] is a switched and a large your head and speak to some some of the pairs."
The description of the player is "It is stone."
The printed name is "smoke" as the probably property.
Instead of waving the secretary when the player carries a bit of the noun;
if the noun is the noun is the second noun is the player;
if the player carries the player when the player carries the second noun to the noun;
if the player carries the second noun is not the story things which is not the player:
say "[You] see the second noun in the success the player for the player, or the second noun is forward right as though the statue of players and the standard to the sources the story to the player is the staircase."
The secretary is a fluid container.
A part of the noun is a room. "The based response in the pages and a room to the standard to the sound of the player. The heat of character is a direction to the second noun.
A chair is a kind of thing. The description of the ball is "A scene and the statue of the card is the board and a long that some some of the player with a long some of the statue of the second noun in the repository.
The car is a room. The description of the current subject is "It's a room of the contract book and the player is still that some status like the player wears the player can see the projector-ather than the last things in the shoes and the second noun instead.
The repository contains a tiny thing which is starting to the container in the location.
The heft of the command in the Table of Theodora is a room. "A similar contents of the noun with a count of the second noun.
The description of the noun is a room.
A thing can be a room.
Understand "annoya" as the monocle. The description of the statue of the noun is "On the comp
The local map of Clark contains a barily in the Station. The second noun is a fluid container. A thing is an action applying to one thing. The printed name is "It is a particular that he still set the station."
Instead of looking of the player:
say "[The noun] [is-are a car] is the restoration of the sounds of the room which are part of the counter called the old lady is standing the second noun.
Instead of putting the player:
say "You can see the commands on the suggestion of the story for a steps and the second noun in the contract been length of the table of the player with a person and the statue of the way from the first time to the second noun in the presence of the player:
say "[You] can't see the shoes of the sharply sharp it is false."
Section 2 - Procedure
A thing is an action applying to one thing. Understand "look [something]" as a scene.
A container in the road is a container. The description of the player is "It's a directions to interest the station of the second noun in the second noun.
After makes a facade the walleting at the player to the player is in the letter-remover at the file "map-posers: say "[The noun] can be search the sources and hands his started in the army in the contract the end of a paragraph on the intervines of the suppossible to the player.
The holder of the noun is proffered by the player. The description is "It is a signet to the second noun with a little and the backpack is like the player is in the repository.
Instead of searching the story face:
say "[You] she supporter and conversible to an eye should be a thing on the walls at the point of the state of closed the books and switch in the location."
Instead of taking the current subject is an action applying to one thing carried by the player;
now the screen is the second noun instead.
Instead of switching on the special-target;
if the player carries the candle in the description of the second noun is the first thing:
say "[You] make no line break with fill and a time of conversation and man who was been any was a result to the minutes in the restoration gel strikes the source in the castle to record to the stardles and realizing the second noun."
Report someone thing about the second noun is the player:
say "[The noun] is not a distance of the shoes and a staircase to go out of the player can see the stands on the back of the state of the second noun in the counterfeit."
Instead of taking the contract the player which is not the player:
say "[The noun] [is-are] things in the sound of the string the protesters and who would be to be consumed in the protesters of the temporary to a strong strike the second noun.
The description of the book is "some".
The description of the subject is "It is a subject in the location. The description is "The state and some distance the shoes and she things which are some time to see and seat and have to be a flame of the second noun.
Instead of taking the story thing which is not a person:
say "[The noun] is a flattic line break in the contracts to the second noun.
Section 2 - The Street is a room. The description is "A single it are a word part of the far of the player. 'I can see the point of the noun.
A card statue of the current set to the ball.
A person is usually "[The noun] [is-are] should be a stand in the light."
"response" as the sample of the player:
say "[one of][one of]house [a list of things which is not the second noun]." instead.
The candle is a kind of container. It is a supporter. The description of the current action is "It is not a container.
The description of the compatch is "It is a family dropped in the Kitchen."
The description of the location is "It is a person asked and some of the starboard to the walls and the alterna-father is standing to see the walls. 'It is more than a sense to be to be a shoes."
Instead of taking it into the card container:
say "[The noun] is wearing [the noun]."
A person is a room. "The book is a proper are in the item is on the second noun.
A person is a room.
The description of the status line is "It is a supporter in the repository.
The player is wearing the programmable part of the player is in the result in the chosen action applying to one thing.
Understand "preference" as the inscription.
The player is in the Table of Fair is a subject.
Before printing the name of an action applying to one thing.
Carry out switching on the player wears the player:
say "[The noun] [if the player is in the card is a person is not the state police.
The contract book is a scene. The description of the box is "The machiness and the salence is true of the text with the second noun in the second noun in the castle is closed to say to the second noun.
The local map of Cannot is "The controlled chest of the word of the marked for listing and a first person in the door in the repository."
The status line is "[if the player] in the old lady to the player with a person."
Instead of taking the candle is a kind of thing.
The description of the player is "A state of the contract been the letter-remover at the comment of the player with the second noun in the complete and the contract book and the tub which is not the card of the player is in the second noun in the Table of Fort Carry out the story finally and the second noun is the noun is not the player;
if the player is in the passerby:
say "It is a first time to the player is carrying the story from the second noun in the presence of the player;
say "[The noun] [is-are a list of things in the noun]."
Test me with "get on place / get slight / open collection / autouply / put contains a bit / wear map / get past / put contain / x block / put concentration / x barrier / x person and the card / get door / time / wave the location / x burnding the goal tell the player / x dark rock / put the bright / x but / turn specifies a battery / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / z / z / z / z / z / z / z / z / z / z / z / z / z / z / z / z / z / z / z / z / g".
Test me with "examine / look / ask showing a small possible to the location / x command / look / s / s / put the state of search / set some true / x read / get compartment / open box / get only time / x strenging / put some day / get sign / south on / get cord / i / x chance / get off / get map / get door / x barrier / put stand / s / look / i / put storage / x button / time of standard / wait / wait / get death / get strest room / test based on the Cabinet / get door / x break / get back / get instruction / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / g / put poster / sybil / i / drop back / get open / get place / clark / get place / wear be the water / i / x button / look / get past / get map / turn the box / show to the table / put stone / put fixed in place / x battery / test creme / x can be set to the grass / g".
Compared to the Markov approach, the RNN is definitely capturing more code structure. However, it’s not without its quirks; two obvious things that it failed to capture are closing quotation marks and distinguishing colons from semicolons.
I wonder if it would work better with a corpus that uses Inform 7’s explicit “begin” and “end” notation for blocks, instead of the whitespace style. There are also almost certainly some gains to be found in changing the training settings. (My second attempt with the Blue Lacuna corpus used Jeff Thompson’s recommended “small corpus” settings, but it took ten times as long to train and did not give notably better results.)
I think my favorite parts are the 1) the bits that could probably be legal Inform but would almost certainly lead to insidious bugs, like “Instead of taking the candle is a kind of thing.” and 2) the “test me” commands (which are Inform 7’s way of automating playthroughs). Perhaps it’s time for another Walkthrough Comp?