In the summer of 2016, hoards of adults and kids in cities worldwide were meandering into parks and dashing down the streets, their eyes glued to their phone screens. They weren’t texting or emailing, but instead searching for virtual Pokémon monsters hidden within their real-world surroundings. Though it passed just as quickly as it appeared, the Pokémon Go craze marked the first time that such a large swath of the general public had engaged so enthusiastically with Augmented Reality—and that interest has only grown since.

Not to be confused with Virtual Reality (VR), which immerses the viewer totally into an artificial environment, AR superimposes images and information onto the user’s physical environment via a smartphone or other gear. Thanks to AR, we can use IKEA’s Place app to drop a virtual armchair into our living room to see if it vibes with the rest of our furniture. Or, at the Intrepid Sea, Air, and Space Museum in New York, we can come face to face with Mae Jemison, the first African-American woman in space, as she guides us through an installation.

With its potential to transform the way people tell stories and interact with their environments, AR is rapidly piquing the curiosity of designers of all stripes. This year, creative coders Zach Lieberman and Molmol Kuo released the Weird Type app, which lets users play with moving typography in space. Swiss digital design studio Milk Interactive AG created an app for the 2018 Weltformat Graphic Design Festival that made posters come to life. Similarly, Berlin-based interdisciplinary design and research agency School of Observation recently rolled out an interactive city-wide campaign, dubbed #TypszPlnsz, with the aim of challenging Polish-German stereotypes that incorporated AR elements.

In recent years, AR has changed from a hobby for the tech-savvy into big business. According to a recent report by research firm SuperData, mobile AR apps are slated to reach $17 billion in revenue by 2021 (if this happens, they’ll surpass revenue generated by VR software for the first time). And while both AR and VR adoption are expected to grow in the coming decade, AR is proving to be more accessible. Apple recently unveiled iOS 12 with an array of AR features. In addition to the release of Apple’s ARKit and Google’s ARCore—two building frameworks to create AR apps—Adobe just announced the private beta of a new tool called Project Aero, created specifically for non-coders. As the technology continues to advance, designers are in a unique position to help shape the future of AR. Will it become a tool that’s useful and engaging, or just another shiny technology that distracts? 

 

Designing New Realities

Pushing AR beyond gaming and entertainment use cases will take the research and experimentation of designers, artists, and engineers alike. One example of such a design studio is the Berlin-based NEEEU Spaces, founded by Raphaël de Courville, Moco Raffael Ziegler, and Javier Soto Morras to help clients figure out how to use emerging technologies to better respond to people’s changing needs. “There are a lot of things to be said about the shortcomings of AR technology right now,” says de Courville, “but you can only find out what it’s good for if you actually try it and design for it.”

One recent AR experiment is NEEEU’s Aero One app, which lets you play virtual instruments and create a symphony of sounds using your own voice. Letting out an ‘Aaaah’ sound into the phone, for example, produces a trail of orange audio bubbles that appear in succession on the camera screen.

However, “playing” the animated harp isn’t quite as simple. It involves holding the phone while repeating awkward, waving hand gestures that would “hit” the strings. While de Courville admits that the app isn’t perfect, he and his team are using it as a means to explore the phone as “a kind of motion controller,” and to learn how to improve user experiences in the future.

Collaborations with cultural institutions offer NEEEU the opportunity to test its ideas in a practical context. Earlier this year, the studio developed a custom AR app for Berlin’s notable Gemäldegalerie in collaboration with museum4punkt0. Its goal was to enhance visitor experience. Given a headset and an iPad with the app installed, visitors were encouraged to hold up the iPad in front of select portraits, activating an x-ray view that showed the artist’s revisions throughout the process.

For another project, NEEEU recently developed an AR prototype alongside the Natural History Museum in Berlin that was tested in a bird exhibit at the Long Night of Museums event. Equipped with headphones and a smartphone, visitors were invited to scan markers on the exhibited objects that would lead to actual sound recordings of that species.

NEEEU’s interdisciplinary team is primarily made of creatives with backgrounds related to design (including graphic design, media design, innovation design, interaction design, and generative design), and most can code. De Courville tells me that picking up the tools and skills to work with AR wasn’t difficult for his team. “An important part of the process was having open-minded collaborators that trust you and vice versa,” says Laurence Ivil, a copywriter at NEEEU. “It’s about bringing together different skill sets and seeing how we can build a product or app that’s going to benefit the environment.”

The Practicalities of Pushing Boundaries

For designers who are used to working in 2D and aren’t well-versed in coding, however, creating AR experiences can seem daunting. Daria Jelonek and Perry-Jame Sugden both recently graduated from the Royal College of Art’s Information Experience Design program and have just co-founded Studio Above&Below, where they focus on immersive projects at the intersection of design, art, and technology. The duo believe that if you have an open mind and a couple of weeks to commit to learning about working in a 3D world, it’s easier than it appears. Although the pair had prior experience with 3D modeling and motion graphics, they credit the internet and talking to people with experience in the field as major resources when learning more about AR.

For their first solo exhibition at SET in London, Jelonek and Sugden presented a piece titled Finding a New Terrain that combined sculpture, print, and AR elements. The idea was to explore how different natural materials behave in an augmented space. Visitors could use their smartphone to see how a block of wood disintegrates into thin air when it’s burned, or what happens to stone if it breaks apart.

“We usually start off with a message and then decide on what we want to augment and how,” explains Sugden. To create Finding a New Terrain, the pair first created the sculptural objects and then used Unity (the game engine that most people use to create AR apps) to develop the interactions and animations that would show its materiality through the smartphone.

While experimenting with AR sounds fun and exciting, how affordable is it for a designer to dabble in this technology?

While experimenting with AR sounds fun and exciting, how affordable is it for a designer to dabble in this technology? Studio Above&Below are currently on commission by the art and technology institution Collusion Cambridge to research and develop augmented reality uses until May 2019, which provides the funding for some of its projects. However, experimenting with AR doesn’t necessarily require an expensive suite: Unity, for example, is currently free to use for those that have less than $100,000 in revenue or funding. And while having an iPhone X is nice if you want to get into the next level of AR experimentation, there are other alternatives, like using a laptop camera.

The Future of Designing AR

AR could become a tool that deepens our engagement and understanding of the world around us, or it could slip into a system that sees notifications, advertisements, and digital detritus constantly invading our view. Avoiding the latter requires designers to consider the social consequences of their product from the onset—which is why it’s reassuring to hear Jelonek and Sugden talk about the notion of “Sustainable AR.”

“Rather than using AR for branding or just for fun, we wanted to promote a new thinking about this in-between space,” explains Jelonek. “How can we use AR to connect people to their natural environments? Or create something that’s good for the planet?” 

AR could become a tool that deepens our engagement and understanding of the world around us, or it could slip into a system that sees notifications, advertisements, and digital detritus constantly invading our view.

Ultimately, when it comes to the future of AR, new opportunities mean new challenges—and not just ethical ones. Designing mobile AR means facing an array of design parameters that break from the well-known territory of the 2D grid; for example, designers have to consider the space beyond the phone screen (or the environments the app will be used in), the scale of objects being overlaid, and the movements users will engage in, just to name a few.

“Right now, designers who want to quickly prototype interactive 3D concepts are facing lot of barriers and steep learning curves,” says Tera Hatfield, design director at Torch 3D, a Portland-based startup that launched a free mobile AR prototyping and design app.

Founded in 2017 by Paul Reynolds, Josh Faust and Tony Falco, Torch 3D wants to make it easy for designers and developers to unlock the creative potential in AR. Natasha Jen, partner at Pentagram, recently joined the company as a design advisor. To find out the pain points for designers and developers, the team at Torch 3D interviewed more than 145 creatives from companies like Adobe, R/GA, and Wieden+Kennedy about their experiences developing AR and VR applications. They learned that the five largest challenges were: learning new tools and needing others; cumbersome communication between designers and developers; finding and creating 3D assets; the amount of time needed for prototyping; and the difficulty of sharing work for feedback.

“Ultimately, production-oriented tools like Unity are fantastic for the final build of a product, but when you’re iterating through a bunch of ideas, it can be very time-consuming and not the easiest for setting up projects,” says Hatfield, who previously worked on AR and VR projects at Wieden+Kennedy as an Experience Design Lead.

Hatfield believes that AR use cases will become more interesting once a greater amount of people have the ability to easily experiment with it. “We need diverse designers with incredible professional backgrounds and skill sets to bring their voice and ideas to this medium to expand it,” she adds.

There’s no doubt augmented reality has the potential to elevate and expand the way we engage with our environments. Currently, designers are largely using AR for playful experiments or adding it on as a flashy feature. However, there are so many possible use cases that designers can explore further, especially in the spheres of education and journalistic storytelling. For AR to become practical and meaningful in our lives, artists and designers need to get involved, collaborate across disciplines, and explore use cases that go beyond pure entertainment.