Augmented Reality (AR) on WeChat What You Need to Know Before Jumping in.

This article was originally published here.

Augmented Reality (AR) is very likely to become the next big tool for brands to gain dominance in WeChat’s highly competitive environment.

Alibaba and Tencent bringing their CNY 2017 red envelope (红包) war into the AR world is a key indicator that we’re facing what could be the next user & consumer experience revolution in China. Plus during the summer both Apple and Google released their own AR development platforms — ARKit and ARCore — enabling developers to quickly and easily create AR-based app experiences. And with iOS11 just released, expect to see augmented reality experiences exploding onto Chinese smartphones over the next 12–24 months.

Although most analysts predict that the enterprise AR market will lead the consumer market in the West, it would not be a surprise if the AR revolution begins with the consumer in China, starting from marketing and social use cases.

The bottom line is that AR experiences are getting easier to create and audiences are enthusiastically engaging with them.

Why should you care? Well — projections for the size of the AR market by 2020 hover around $90 billion, while the expected number of AR-enabled phones in 2018 is 900 million. Within both estimates, China accounts for the lion’s share. Of the 100 million people Gartner expects to be shopping in augmented reality by 2020, a majority will be Chinese. AR is also expected to play a key role in the current Alibaba-led move to merge online and offline commerce (see their acquisitions in traditional retail), resulting in highly integrated purchasing experiences.

Are you potentially at risk of missing out AR opportunities to engage with your Chinese consumers now? If you’re a consumer-facing brand with a large audience, then the answer is probably yes. The initial novelty of AR will offer excellent virality to the first wave of campaigns and experiences, but as with most new interaction innovation, the novelty effect will die out quickly.

So where does the overall WeChat AR experience currently stand in this revolution?

• Limited to browsing AR-enabled web pages
• Still far from the full capabilities of modern mobile AR (objects, depth, surface, positioning and lighting condition recognition). In other words, limited to Location Based AR where an object floats in 3D space regardless of other objects. At least for the average brand (more on that below)
• Currently limited to WeChat on Android devices — on the iOS version we need to prompt users to open the AR experience using Safari (might be fixed soon)

WeChat as of now already offers a couple of AR features built into the app: real-time translation, street view, book cover analysis.

ChinaChannel

But China’s most ubiquitous app makes up for its limitations with its 900M+ user reach, of which 80% access it on Android phones. Since WeChat essentially facilitates an AR access point in almost every smartphone owner’s pocket, brands can reach a large portion of their audience through AR campaigns on the app. This already makes AR far more viable than VR for mass consumption potential.

It’s true — there are still significant limitations to WeChat AR, but there are also workarounds. And as tech in this area improves over the remainder of 2017 and into 2018, friction will decrease and power available to creatives will increase. In WeChat’s crowded marketing space, a whole new arena is being born.

In this article, you’ll learn about the two main types of AR and the current state of evolution for each technology. Then, we’ll look specifically at what’s possible in terms of AR on WeChat. Finally, we’ll venture into the future, and look at what the world of augmented reality in China might look like between 2018–19.

Think of current day AR, and Pokémon Go is probably the first thing to spring to mind. Downloaded over 500 million times to date, the app was one of the most viral topics in 2016. But there’s the catch, Pokémon Go needs to be downloaded before you can ‘catch ’em all!’. Which brings us to our first distinction:

Native AR vs. Web AR

Pokémon Go, or Snapchat’s ‘Lenses’, are good examples of native AR. Native to a downloadable mobile application, these kinds of experiences are powerful in combining fantasy with reality with recognition of complex environments encompassing objects, surfaces, textures, and lighting. On the downside, having to download the app creates friction for the user. The advent of ARKit and ARCore allows for easier creation of native AR experiences — giving more control to creative talents and decreasing their reliance on ultra-specialized developers.

In a nutshell: More suitable for frequent-use experiences that require full modern AR capabilities and provide enough user value to be worth downloading a >50Mb app. Think games like Pokémon Go, tablet-based advertising experiences, or IKEA’s ‘preview’ catalogue app, “IKEA Places”.

On the other hand, Web AR experiences are accessed by simply opening the Internet browser on your phone (or WeChat! more on that later). These experiences are limited in capabilities compared to native AR. First, they need a marker- a physical symbol in the surroundings for the phone to recognize and trigger the information overlay. Secondly, they are not capable of depth perception and have limited object recognition capacities. But these limitations are being resolving fast. And with no need for an initial app download, Web AR offers an exciting prospect for almost frictionless and highly shareable AR experiences — with just a single touch or swipe. Apple’s release of iOS11 served as a real game changer in Web AR with Safari finally allowing in-browser (WebRTC) use of the camera feed. In short, this update brings AR to every recent Apple smartphone in China.

In a nutshell: perfect for simple and lower-frequency usage, such as O2O experiences designed for one-time marketing campaigns, events or visits etc.

If you’re on desktop, scan this QR with your Android phone to see Web AR in action

How Can You Use Web AR Today?

Web AR can be a great way to engage with consumers in a new dimension with low barriers to entry. Here are a couple of key use cases where it can be used in the marketing sphere:

Augmenting an Offline Experience with Interactive Information Overlay

A print ad for a new car model that allows the user to view the car in 3D, customize the ‘AR car’, and then book a test drive or make a purchase.

A portal door that allows a glimpse of a virtual world when opened.

Marketing Use Cases: Print advertising, mall advertising, event interaction, live 3D action with products, real estate, etc.

Virtual Fitting: Trying out Items and Item Variations

With an AR marker around the user’s wrist, they can virtually ‘try on’ a new watch through their phone.

Marketing Use Cases: Accessories, clothing, furniture, etc.

Social Engagement with AR Games

Left: Piloting a virtual flying car with a hand sensor. Right: Social mini basketball game

Marketing Use Cases: Brand awareness & social engagement campaigns.

Assistance

Draw any picture on a piece of paper perfectly, just by following a projection of the picture made by your phone

Marketing Use Cases: Product usage guide, assembling assistance etc.

Web AR Through WeChat: Limitations & Workarounds

Above are just a few examples. Yet, common in all of them, there’s still a number of limitations to bear in mind when it comes to today’s web AR capabilities:

• The required use of markers as real-world reference points
• No significant environment analysis capabilities
• No dynamic lighting interactions, unlike native AR
• Significantly lower performance against native AR

In addition to the above limitations, Web AR abilities through WeChat’s browser are still very much in the experimental stages. Here’s what we know now:

The good news is that there is a workaround. If Web AR faces any issues in WeChat, whether on Android or iOS, it can be detected and we can prompt the users to open the Web AR experience in their default mobile browser.

Directing the user to open the experience in the default mobile browser takes 2 additional taps.

AR in the Short-, Medium- and Long-Term Future for WeChat and China

iOS Safari Web AR to Reach 90% of Compatible Apple Devices by End 2017

Based on iOS10 reaching 50% penetration within a month of its release in 2016, expect that 90% of compatible devices will be running Apple’s latest OS — that enables WebAR in Safari — by the end of the year.

iOS Web AR in WeChat Too?

Apple’s decision not to open WebRTC to WebView is probably a move to avoid potential cannibalization of their native ARKit. But no doubt behemoth third parties such as Tencent will pressure Apple to allow their apps’ browsers access to camera and audio feeds. We believe there’s a moderate possibility of this limitation being removed within the next year- meaning Web AR will be accessible directly through the WeChat browser on iOS.

Android WeChat Web AR to Improve Steadily

On Android, the X5 browser’s performance will continue to be optimized with the growth in number of experiences released and developers improving the software libraries available. The focus in the near future will be on enhancing performance levels to deliver better refresh rates.

WeChat to Offer Dedicated AR Features to Selected Brands?

We’re currently witnessing an explosion of close collaborations of B/A/T with major brands, extending them preferential access to exclusive or unreleased capabilities. We believe this trend is likely to continue with more opportunities for multinationals and major Chinese companies to leverage ties with Tencent to access WeChat’s app-specific AR capabilities.

WeChat to Release a native AR API Open to Third Party Developers?

Although Tencent is allegedly abandoning its own QAR platform project, we believe it is probably a strategic move to avoid having to build and maintain their own native AR framework up to the benchmarks set by Google and Apple. Instead, the priority may have shifted to integrating ARKit and ARCore SDKs as a base for AR projects. The strong ambition Tencent has shown by venturing into AR indicates that a third party-enabled AR platform is still on the table. Especially since it would allow mutualizing recognition capacity across a wide range of use cases, in a privacy/data sharing trade-off with brands and businesses using the platform.

Presentation of Tencent’s QAR platform capabilities and integrations at the SIGGRAPH conference, December 2016

And there is a good chance Mini Programs will play a key role in WeChat AR capabilities as they allow the harnessing of device processing power over that of browsers, making for smoother AR experiences.

Conclusion: Should You Go for Web AR Through WeChat Now, or Wait for the Dust to Settle?

It can be difficult to judge when the time is right to jump on the Web AR bandwagon. Now? In three months? In six months? Here’s our simple guide for figuring out how soon you should make a move:

You should go for Web AR on WeChat now if…

• Using a marker as a reference point is acceptable
• Using simple 3D shapes & animations is sufficient for your intended experience
• Your target audience is B2C, with a majority owning Android devices
• You can accept redirecting iOS users from the WeChat browser to Safari

Wait a little longer for Web AR on WeChat if…

• You can’t or don’t want to use a physical marker
• You’re targeting a broad audience across varied device types
• You need to include complex 3D animations & objects
• Your user flow strictly requires staying in the WeChat browser
• Your experience needs a very reliable tracking capacity

It is possible that your creative or technological requirements may demand too much of current technology. Nonetheless, major improvements are on the horizon in the short-, mid- and long-term future — so stay updated! With AR becoming available to mass audiences, the WeChat marketing landscape is likely to change dramatically. First-mover advantage is up for grabs for those who can act fast at the right time.