(September 25th, 2020)
TL;DR
This exposition relates augmented reality to street art to the resonance of spatial constraints. There is a narrative arc that is best experienced reading it in full, but I understand that attention is scarce and so here are some teasers / key takeaways:
- We have the tools today to bring portraits to life similar to the portraits in Harry Potter. We can make them speak any text input with any voice, and we can give them backstories that they use in those conversations. For examples, scroll to the end.
- We explore the history of street art and then ask what would happen if every mural portrait was brought to life as an interactive experience? What would artists do with that and how would this network of characters change our relation to the physical space within which they exist?
- We posit a 2x2 matrix of physical/digital on one axis and objects/map on another. For example, physical objects on the physical map could be buildings constrained to their location and digital objects on a digital map could be websites on the internet. The ideas to which we arrive in this piece are digital objects on a physical map, a largely unexplored square. This is discussed here.
Listening to a place
In How to do Nothing, Jenny Odell discusses Mastodon, a decentralized version of Facebook, in the context of finding a network that lets us “listen to a place”. She doesn’t find a suitable solution, but it reminded me of something specific - the picture portraits from Harry Potter. These portraits were living avatars of deceased people that were prominent in the place where their portrait remained. Former headmasters of Hogwarts School of Wizardry resided in the headmaster’s chambers. A well-liked head of a wizard family might reside in their family’s ancestral home. These portraits all had specific and consistent spatial coordinates and were geofenced to that place. A portrait headmaster of Hogwarts could visit other parts of Hogwarts, but could not go to another school like Durmstrang.
I see this apparent limitation as an advantage. A student walking past Dumbledore’s frame might say something of interest and portrait Dumbledore remembers this. Later, another student speaks to the portrait and Dumbledore shares a tidbit from the prior exchange. That the frame is spatially constrained means that it attunes to that area. That one has to visit it to chat means that those interactions are temporally constrained as well. All in all, you could interpret that this network of portraits allow us to “listen to a place”. Analogues in the real world include the bartender at the local pub and the hairdressers at the salon. These roles have existed for a long time and they serve as push-pull mechanisms for “listen[ing] to a place.”
It is instructive to think about how spatial context changes how we listen and what we hear. When we are in an office space, we have context that this will be a meeting among a small group; the discussion can hone in on minute details and we focus on what is important in the space between just these people oftentimes with a focus on work. When we walk in Central Park, even if the group is the same, the atmosphere changes the conversation; we now observe our surroundings and conversation becomes less about minute details and more about expansive thoughts. This isn’t an outside versus inside distinction either - a lunchroom would change the conversation the same as would a trip to a museum or a side street in Boston. Spatial context is important to the communication protocol that develops among the group. It adds shared minutiae to the conversation, and its absence is a palpable strain on internet or video discussions. In NYC, I walk around the city and see protestors and street art and people eating pizza on the sidewalk in front of a pothole and notably do not see the farmland or midwestern twang that I might get in Kansas. Both myself and the Kansas farmer might have a pothole, but when we come together on the internet, we first have to develop shared context before we could reasonably think that we understand each other and how we prioritize the issues.
What would we hear from a portrait named Bob chilling in a Central Park tunnel and how would that differ from portrait Jakob in Berlin on Oranienburger Straße? Bob would hear from tourists, from older residents of the UWS and UES, from people walking dogs, from buskers on bright spring and crisp fall days, from the pigeons squirreling for food and the squirrels digging up saved morsels, and would go to sleep when the sun went down. Jakob would be witness to refugees and people dancing in the streets and street artists and he’d be awake during the night energy just as much as during the day parade.
If Bob and Jakob were twins split at birth, the stories that both would witness would change them irrevocably such that upon meeting one day in a neutral frame, they’d be wholly different characters. In short, they would represent the place. Or at least they would be reflections of it. Talking and listening to them is listening to the place.
But not quite. Conversations with B&J would miss the observations that we glean when we wander through an area and have all five senses working in congruence. Bob and Jakob receive visual and auditory input but are without touch, smell, and taste. What does the area feel like? Is the rain so cold that it tempts snow? They would not know. Is there lavender blooming nearby and filling the area with a pleasant scent? Unless someone told them this, they would not know that. And even if someone told them, their perspective of the experience would be limited. It’s similar to how a hairdresser or bartender witnesses the social waves in the water, but rarely notice the roses outside the shop with the same regularity. The rose is still the rose, but its observational absence changes the story. Imagine listening to a forest without any birds. What you hear won’t be startling until it very suddenly is concerning as it dawns that there are no chirps or birdsong.
And what about the coherence of those senses? When we breathe deeply and then shutter our eyes repeatedly, multiple input streams enter our conscience. It is only in combination that we feel them for what they are. Watching a desert scene in a movie doesn’t impress the same feeling as being in the car driving into the desert. This remains true even if we put the movie theater in the desert. The challenge is not in representing the desert visuals. It’s in the serene almost-silence; it’s in the occasional gust of wind; it’s in the sand pebbles that annoy us but are reminders that our soles should be listening as well. It’s in all of this and all at once.
This abbreviated sensory input isn’t what I want for my experience of a place, but these shortcomings are reasonable to what we should expect when a portrait representing a dead person is magically gifted s.t. it persists as that person. We attribute to these portraits vision, auditory, insight, memory, and logic, and we refrain from olfactory, touch, and locomotion. Scent seems possible but locomotion and touch don’t because they violate some part of the frame constraints. The result are characters that are halfway there and promise us an abbreviated record of the place. This reminds me of street art.
The trademark of street art is that it’s subversive (“guerrilla art”). It’s also people expressing themselves and telling a story of the place. It’s people reclaiming the space from the corporate colonization. When I see street art, I think about how the building it’s on, whether it hauses squatters or corporate raiders, has been granted berth to tell a story founded Here; Now. Within a major city, the network of street art is a record of what territory were fighting to maintain their identity and hadn’t ceded their land to the Appropriate Police. And each individual piece is a character representing that Here; Now. We’re going to grant life to those characters and imagine that a single street became host to a litany of interactive portraits, each with their own names and histories and voices. But first, let’s get on the same page as to what is Street Art really.
Explain Like I'm Five: Street Art
There’s an art called graffiti. It’s commonly text and the artists all have their own style. Sometimes, it includes cartoon characters too. It dates back to 1960s Philadelphia when this guy CornBread tagged his name everywhere because he wanted to show people that he was alive and not nameless. Mostly streets sure but he also tagged an elephant at the zoo and the side of the Jackson 5 jet.
In the 15 years afterward, graffiti spread all across New York City. Two major reasons why were that the subway reached a majority of the city and that the city couldn’t afford to pay for transit maintenance. The artists liked seeing their name and work flying everywhere in the city, and graffiti was a way for people to advertise themselves.
The city did not like this, and Mayor Lindsay began the first war against graffiti in 1972. He arrested artists as vandals but failed to stem the art. In 1982, the Broken Windows philosophy became popular. It said that if there is a single broken window or car left in an area, then all the other windows and cars will soon be broken. The first example is a sign that people don’t care, and so breaking more costs nothing. It’s fun, so why not. This is true - it is actually a lot of fun to take sledgehammers to old computer parts.
With that idea in mind, the city increased police and security on the subway cars. In 1984, they initiated a policy to get even realer. That policy actually worked and the number of graffiti artists in NYC shrank. Some went to London and other parts of Europe and set up shop there, which was actually a great result for popularizing graffiti. Others turned towards new targets like property and roofs before covering the streets like you see today.
The local communities didn’t like that because the graffiti from the subways already had a really bad connotation. People wanted this cleaned up and the costs were exceptionally high. Throughout the 90s, NYC mayors like Guiliani instituted laws and efforts to end graffiti. Those laws were picked up by other cities across the nation. But the real real is that that won’t get rid of graffiti because it is at its core a counter-culture expression; being illegal carries with it a risk that the artist embraces.
The laws have pushed graffiti artists to become legit and —> street artists. There is a fluid idea behind both of these as “art on the street”, but Street Art is often accepted in advance or even paid for by a client. It is not always true. Banksy, for example, doesn’t ask permission to spray his famed pieces. The ironic part of his murals are that the reactions are flipped; the people who like it are the public and the people who don’t are frequently his fellow street artists. This is because he draws people to the local area and so it makes bank for the area no matter what his art is expressing. That he can make something that is controversial and people say yay because $$$ is maddening for other artists.
The Story of 5Pointz is instructive re drawing people into a local area with street art. This rich real estate developer Wolkoff has a building in Long Island City (Queens, NY). Beginning in 2002, and with Wolkoff’s permission, street arts cover it with strong work, all curated by a legit street artist named Jonathan Cohen. It was so good that it became a worldwide mecca.
In 2013, Wolkoff wants to destroy the building and rebuild a new set of apartments targeting the new and mostly white residents to LIC. This ball is rolling when one night, without warning, he has the place whitewashed. This was not received well, and the artists sued him. Without getting into the case too much, all the appeals were finally finished in Feb 2020 and it was decided that he has to pay $6.7m to 21 of the artists involved, setting a precedent because it used the Visual Artists Rights Act and suggests that individual pieces of street art could be federally protected.
In the aftermath of this event, Wolkoff tried to use the 5Pointz name and people with cred collectively said “Nah”. That cultural appropriation is a real sticking point though for street artists. Where the art is prominent, it either lowers property values because it’s seen as a blight (graffiti) or it greatly increases long term value because of its effect as an initial beacon for gentrification (street art). Both stories are tough spots for the current residents and invite lots of questions about the role of street artist as a public good. I think this article says it well:
Long Island City is one of the most rapidly gentrifying neighborhoods in New York City. But it is artists, like those that popularized 5Pointz, who brought the neighborhood attention and raised its value. Once they had served their purpose, they were disposed of in favor of condos for the rich. [Street Art and resulting affluence] attract property developers who permit graffiti within certain parameters that serve their interests. Wolkoff promised to maintain significant wall space in his new development. How much space? Who will be allowed to paint? What will be the criteria? These questions fall on deaf ears and we are reminded that we are lucky to be getting any space at all.
Cities would 100/100 prefer street art to other crime, so places like 5Pointz, like Joe Ficalora’s Bushwick Collective, these places serve as invaluable safe spaces for the community and entry points to art & culture.
Street Art & Value
In How to do Nothing, reflecting on the Rose Garden, a public space in Oakland built by the Works Progress Administration, Jenny Odell says:
those spaces deemed commercially unproductive are always under threat, since what they “produce” can’t be measured or exploited or even easily identified - despite the fact that anyone in the neighborhood can tell you what an immense value [they] provide.
We saw in the previous section that this is true for street art. In contrast to decaying or barren building facades, a mural of two hands forming as one or a sign mocking advertising and consumerism beautifies a neighborhood. It is also a positive example to future generations of artists. The neighborhood wants both of those and yet, similar to the park, the art is commercially unproductive by itself.
Could street art be commercially productive? We have one approach where the artist is hired to draw a mural on a facade. That exchange is transactional and serves only intangible future commercial value vis a vis attracting people, serving as a design centerpiece, and preventing graffiti. Residential buildings are paying customers, as are restaurants in Little Italy (LISA project) and the LES gates (100 Gates).
But hold up. Do people even care about the street art? There’s some people who really care about it. The artists do because it’s their craft and how they express their name. When regular joe is walking down the street though, he pays it less attention than the atm, starbucks, or his favorite food cart. Why is this? Those three examples - the atm, starbucks, and the food cart - are actually very much like street art in that they are spatially constrained. The atm and the food cart are also similar in that they are outdoor “machines”. However, the key difference is, once again, that they are commercially productive. In China, this is even more true where all kinds of businesses are spatially-constrained outdoor “pop-ups”.
Here’s an idea for getting Joe to care about street art. Similar to the food cart, the atm, and the Chinese stalls, just have the street artist set up a shop at their piece. Artist makes a dope mural on the side of a wall and then, just like the vendors in China, puts a QR code next to it that reaches their store. They sell posters, shirts, and masks from this store, with their work serving simultaneously as expression, beautification, and as advertising for their own livelihood. This subversive act marries the commercial productiveness of spatially constrained “machines” we care about with the artistic appreciation of the art.
Yes, the QR code diminishes the first two experiences, so remove it and instead assume that there is another way to pair artwork with artist with web-accessible mobile device (like an app). This lets us do a lot more with the interface and guarantees trust in the experience. It also has a big con in that QR codes are catching on now.
We’ll talk more about the interface later. More pressing is returning to the question of commercial value. At this point, we actually have something that has concrete commercial value - either a piece brings in $$$ or it does not. This becomes a lot more interesting when we recognize that there are no conditions on what a store can do. We can generate any storefront on top of the underlying technology, with the main difference being that the access point is not a website but a physical presence. It generates surprise and joy behind each piece. You see one and you “click” it and you get:
- The artist storefront similar to all the others we find on the web, except now you are directed to exactly where the artist wants you to go without blindly searching for the piece on google or first uncovering the artist’s name. This could be a store or it could be a single unique item like Air Banksies.
- Maybe the artist doesn’t want you to go to their store, but instead wants you to support a non-profit for aiding poverty or improving water access. They send you to a page that says why and then a link to donate or volunteer.
- By clicking through, the audience signs up for the next experience, and that extension of the artist’s expression can be broad. They could play music through the device or a nearby sound system, suggestive of being the medium through which an experience is channeled like Bon Iver coming out with a new album. They could send the audience to another nearby piece with instructions for what to do there, a treasure hunt that invokes community or exploration. They could request that more people join before a game is unlocked, which could draw nearby people together and create a novel experience (like Killer Queen).
The above are considerably different than what would likely be explored at steady state, but they demonstrate what augmented reality entertains. Once there is an expectation of the art unlocking a new world, artists can build for that expectation and have their name known through that channel.
We now have a third avenue towards commercial productivity. This is the ability of the experience, and not just the art, to draw people to an area, keep them there for a prolonged period, and then tell their friends about it. It is an intangible value, but one that a local community can rally around because it increases commerce when there is more well-to-do foot traffic. That’s the premise upon which Pokemon Go (and parent Niantic) is built, although they do it by explicitly leading people to areas that have stores with whom they are affiliated.
The Life and Death of Great Street Art
I left two threads dangling that we’re going to reconcile here. The first was about granting life to a gallery of street art portraits and the second was about the interface that connects people to the artist through the piece. Recall Bob from Central Park and think of him not as a thought exercise but instead as an actual character in a mural on the side of a wall.
We can today animate Bob to speak text with any voice. Lagging behind, but also starting to look possible, is the ability to give him a history and consistent character profile, along with conversational memory. Through our pocket computers, we can know where the user is and in particular which character they are near. We can see through the camera what they are looking at in case we haven’t seen that particular character before. This means we don’t actually require ugly QR codes to get this to work but can work with these other primitives instead (through an app). Then the user can speak with the character they are near. More poignantly, Bob can learn from the user and grow into the character that the Place defines.
Bob isn’t limited to just a text input and a video+audio output though. He is in a geofenced area and knows about that area because Bob has eyes and ears. So he has a sense of who are the other characters in the hood and can talk to or about them. This is possible because he inhabits the same digital world overlaid onto the physical world as they do. He can use those same eyes to watch through the camera and see how many people are in your crew. So he has a sense of if he’s talking to a lone traveler or a crowd of giggling girls. Bob can watch the sun pass the hours, so he knows the time. This gives him insight into what is happening right now locally. Bob has a lot going for him when it comes to being the base of an interactive experience upon which the artist can build to further enhance the story.
And then, like all street art, Bob will one day pass. They can live for a long time or they can abruptly depart, but it’s never forever. In his stead, something else may appear or it may not - it’s dependent on the neighborhood that they reside and the whims of the artists. And that’s … that’s fantastic. It means that there is a cycle to representing the place. At the end of that cycle, as the place has said everything it was going to say, something new will grow when the phoenix that arises in its stead speaks again. When that happens, Fred will emerge and we can host Fred in this same interface.
The innovation is that these are consistent digital constructs on the real physical map. We already have physical constructs on a physical map - those are buildings. And we have digital on digital with the internet. We also have physical on digital - see onlinetown or secondlife. Putting digital on physical is mostly unexplored as far as I can tell. It says that there is a location in the real world that we must visit in order to interact with a non-physical entity. This is not a bijective map; while a single platform can have limited entities in any one location, a new platform can make their own and have them be non-compatible. If you’re having trouble imagining what this gains us over digital on digital or physical on physical, here are a few examples:
- The menus in a restaurant are digital constructs associated with physical ones (the restaurant). This is a major reason why they are all converting to using QR codes where they can be easily changed to be a new menu as seasons progress.
- The information for an airbnb stay or coffee shop like passwords, how the machines work, checkout instructions, what is the style of the curtains and where to buy them, etc, are all digital constructs associated with a physical place.
- Visiting a piece in a museum is only one part of the experience. There is a wealth of understanding behind that art, all of which are digital constructs tied to that piece and its physical location.
Pokemon Go is also an example of digital on physical, but whereas all of the above have consistent digital constructs, the pokemon do not. They also don’t have stories, nor do they last beyond a few hours at that location. By having the digital constructs be consistent on the physical map, we get the expectation that the character will persist in that place. That’s important! It means that we can return to the character, that we can tell our friends about them, and that we can associate with that location a spark of joy, which is an important element of game design.
Do artists now have incentive to design with interactivity in mind? Yes, I see that as the end result of this direction. Providing a reliably good platform that can turn artwork into living characters would encourage a lot more than just enriching a few of today’s murals. Rather, it would push towards a future where people create with this in mind. That suggests a world where artists are designing organic public spaces with characters that can grow alongside the people who live and play there. It’s not lost on me that this change in the public space would portend a different future which further strains our attention mechanisms. Odell talks in her book about a capitalization of the self by the twin ideas of productivity and efficiency, “One might say the parks and libraries of the self are always about to be turned into condos.” This concern is founded because alongside Bob and Jakob will be a panoply of members from the family of starbucks sirens drawing us to coffee or ronald and his clown of misfits sparking diabetic joy in our children’s fawning hearts. I am not sure how to square that circle but leave it here for posterity.
A Night at the Museum
In the prior section, we included the museum art as an example of digital constructs on the physical map. There is an opportunity for an experience inside the Met where the character in a piece or the artist behind it speaks from their perspective. The two paintings below are being driven by just audio and not video. It is without precedent to be able to talk to Rembrandt or Jacque Louis David about their experiences or what they think of the hashtag pandemiclife. Also without precedent is the opportunity for the museums to network their collection of geofenced characters.
These could be educational, but they could also be just engaging and interactive, built by the curators to facilitate higher appreciation for the museum or a wing. And of course we can do this with sculptures as well. There is even more room for interaction with sculptures because we can yield the 3D mesh much more easily and play with how those characters move through the scene.
I haven’t spoken to enough curators to know if museums want this. However, I’m in like with the notion that this can all be done subversively and without their permission.
If something here hit a nerve, reach out to me on Twitter.
Much appreciation to Adam, Dino, Lucy, Michael, and Ren for discussions and feedback on this post.