Short tweet version of this is here.
Why
For a long while now, I’ve wanted to feel what it’s like to live in a world where the inanimate becomes alive. What does the street art on the corner say? What about the museum? Or the paintings inside the museum? I asked this question more deeply in Space and Augmented Reality.
Utility
With respect to utility, think of this as a federated approach to getting information where, instead of there being a website that we get to via searching the internet or Google maps, there is instead a character that we can talk to that represents that information. This character is up to date because it doesn’t just live near that place - it represents that place.
This is very similar to the utility of the portraits from Harry Potter. They were very powerful figures for pushing along the story and interacting with because they were there, readily receiving information, and able to communicate what they knew in a conversational manner.
A modern example might be the Metropolitan Museum of Art (”Met”) knowing its hours, exhibits, and paintings, and consequently being able to talk to anything going on inside of its hallowed halls, as well as interjecting with worldly knowledge. Similarly, the paintings would be their own characters, with either the painting (Mona Lisa) or the artist (Monet) representing the work and talking with the audience about it.
Art & Entertainment
Another angle is entertainment. Imagine walking into a museum where a litany of characters, all localized to their physical representation, were alive and talking about each other and the world. They would foment rivalries and form cliques. A great game designer could do a lot with this.
With respect to art, we’re describing a way for it to live on forever. The artist could guide the experience or just let it go and learn on its own.
Let’s build it
I decided to build a first version. I think the ideal version involves vision algos so that the characters actually talk in a communicative way, but that would take too long and I wanted to feel what this was like without all that.
Mobile App
So I built a mobile app in Flutter. It’s powered by a Golang backend talking to OpenAI’s API to get responses and a frontend character creator. Every night, there’s a cron job that runs to retrieve events from created Characters that have events (e.g. The Met), so it knows about them. Here’s what the main screens look like:
Clicking on a character on the map or a chat with that character takes you to a screen where you can chat with them. This talks in real time to the OpenAI API, fueled by some prompts I made for this purpose. Here’s an example of that with The Met and MoMA. The prompts know about their events.
Creating Characters
I also built a character creator to make it easy for myself and others to make them. You can find it here. Feel free to create any one you want. My friends made some.
What did I learn from living in this world?
There’s nothing really engaging about this yet. It’s a ton of fun to build this technology, but there’s no real world problem that it’s solving [well] and there’s marginal entertainment value given the state of the tech without tremendous investment. I think this is the same problem that Pokemon Go ran into with its AR component, as well as all of the other games it tried to create.
On another note, I initially also built this in a way that every user had their own Familiar that interacted with the world for them. Your Familiar would go and talk to other Familiars representing users as well as static location characters, then come back to you and have information. Say you wanted Mexican food that day for lunch, you could tell your Familiar to go do this and then it would garner a bunch of places, talk to them, and then get back to you with information as well as easy access to those chats for you to pipe in.
Yeah, so this needs a LOT more polish than what I budgeted for it to be good. Instead, my Familiar would just go off and have inane conversations with the other Familiars that I needed to then scroll past. It would even try to buy things and they would haggle with the cost always being $XX.XX.