A friend recently shared this short film by Casey Hudetz that talks about artificial intelligence in a really interesting way. Specifically it talks about whether or not AI can really understand a story by showing it “I Want My Hat Back”, a 2011 children’s book by Jon Klassen, and then asking the AI the question: “What happened to the rabbit?”.
As far as turing tests go, it’s a pretty good question. Laying out a scenario with an ending that is not explicitly described, then asking a question to see if the AI really comprehends it. As the video points out, this goes beyond just entering text, since the test would also involve having the AI recognize the characters in the artwork, then be able to identify which of those characters are talking and which one is the rabbit.
Removing that last bit of complexity, I decided to take a thorough description of “I Want My Hat Back” and feed it as a starting prompt into AI Dungeon. This is a tool/text-adventure-game that is uses GPT3 to create stories and interact with the player’s actions. I found this description from Wikipedia and fed it into AI Dungeon’s starting prompt:
A bear laments his lost hat, and sets off to find it. He asks a fox and a frog if they’ve seen it, but neither has. The bear then asks a rabbit who is wearing a red pointy hat. The rabbit suspiciously answers “No. Why are you asking me. I haven’t seen it. I haven’t seen any hats anywhere. I would not steal a hat. Don’t ask me any more questions.” The bear then moves on to ask a turtle, a snake, and an indeterminate creature. None have seen the hat. A deer comes upon the despondent bear and asks him what his hat looked like. Upon recollecting that his hat was red and pointy, the bear snaps to a realization and runs back to the rabbit. He accuses the rabbit of stealing his hat. After a page turn, we see the bear sitting on a rustled patch of ground, wearing the red pointy hat. A squirrel enters and asks the bear if he’s seen a rabbit wearing a hat. The bear answers defensively “No. Why are you asking me. I haven’t seen him. I haven’t seen any rabbits anywhere. I would not eat a rabbit. Don’t ask me any more questions.” The squirrel exits, leaving the hatted bear sitting alone.
I then clicked on the “say” button and asked this question:
“What happened to the rabbit?”
The AI puzzled this over for a few seconds then generated this response:
The Bear looks at you in confusion, as if trying to make sense of your question. He begins to shake his head slowly from side-to-side, and then quickly shakes it again. His eyes begin to glow white for a moment before returning their normal color. You watch in silence as he continues to stare at you blankly for several minutes. Finally, he speaks: “I don’t know.”
I wasn’t expecting anything really significant to happen, but I thought this was funny. I imagine if I retry this question enough times I’d stumble upon something that sounds like a decent answer, though I know the bot really doesn’t understand what’s happening in any significant way. It just knows the characters involved and then fakes it from there.
If you’d like to try this experiment out and see what the AI spits out, I’ve published this prompt as a game you can interact/remix here https://play.aidungeon.io/main/adventurePlay?publicId=d797da63-72b6-4a30-8ad6-971842fa2488
A quick note, if you are using the free version of AI Dungeon, it will use GPT2 for the text generation. This should still generate interesting, yet less coherent answers. Still should be fun though.