Welcome to the Metaverse

Last week, Facebook held its highly expected keynote, Facebook Connect 2021, and confirmed the firm’s new direction. Rebranded as “Meta”, the social network parent’s company will from now on be betting on the future and investing in the metaverse.

Software, investment, education, AR & VR equipment… Nothing is left to chance when it comes to supporting the birth of a whole new world.

In early September, Meta (previously known as Facebook) released camera glasses in collaboration with Ray-Ban. With their 5MP integrated camera, they not only allow you to capture photos and videos to share your adventures, but also listen to music or answer phone calls — so you can stay present with friends, family, and the world around you.

Named Stories, these smart glasses are just a first step for Meta toward building full-blown AR glasses. 

During Facebook Connect 2021, project Nazare was also confirmed. If the company stayed really evasive on their augmented reality glasses project, it is clear that they will be an additional tool to bring the metaverse to “our physical life”

Simulation footage – courtesy of Meta

From what Zuckerberg said during the keynote, their main challenge and focus actually are on delivering AR glasses that people will be able to wear in their daily life, and therefore to succeed in embedding all the necessary technology (holographic displays, projectors, batteries, wireless adapters, chips, cameras, speakers, sensors and more) in glasses with a thickness of about 5 mm.

Meta also presented several augmented reality demos based on its research and targeting this possible product. One of them is simulated but reproduces an experiment the company is really working on. It shows a group of friends who are organizing a “virtual party”, all together around a table but each at home, via WhatsApp.

Simulation footage – courtesy of Meta

If the Nazare project aims to create glasses that can be worn all day and that display virtual elements on transparent lenses, Meta is also interested in augmented reality by “camera passthrough”. That is to say that the user wears a virtual reality headset whose cameras film and transcribe his environment on screens. This will be one of the main features of its next high-end helmet, dubbed Project Cambria. Its release is scheduled for next year. In the midtime, Facebook is adding a mixed reality platform to Oculus Quest. Named the Presence Platform, the toolkit is an evolution of the Passthrough API and should allow developers to deliver much more compelling mixed reality experiences. Meta presented it as the key to feeling connected in the metaverse.

But just as the hardware continues to be refined, the software must continue to evolve too.

To do so, Mark Zuckerberg announced that he plans to expand the capabilities of its Spark AR platform to give developers more tools to build experiences that leverage existing smartphone-based AR and the company’s upcoming AR glasses.

Facebook’s Spark AR Studio toolset was initially built largely around AR face filters but has been steadily expanding with more features. The latest announced features aim to improve the platform’s understanding of things beyond the user’s face: hands and body-tracking and geo-awareness will soon be available on Spark AR Studio.  

Users will soon be able to map effects to 20 different 2D key locations on a body using Spark AR Studio’s new body tracker and apply them to a single person, numerous persons, or isolated parts of a body in a single scene. 

The improved hand tracking will also allow you to play with AR interactions even more. Meta said that this will be the first function of a “multi-phased capability rollout” which will add additional hand-tracking features (finger-tracking, grabbing…) in the future.

The updated toolset will also soon support geo-mapped experiences for public spaces which will allow for location-based AR experiences that will be accessible from specific locations. This could enable endless location-tied experiences like discovery tours, scavenger hunts… Lately, all the AR actors took interest in geographic integration through anchors (Microsoft with Azure Spatial Anchors, Apple with ARKit Location Anchors, MagicLeap with the Persistent Coordinate Frames…). Attaching AR experiences to geographic locations is part of the most popular feature for users.

Simulation footage – courtesy of Meta

Finally, Meta is trying to expand the virtual object pipeline in Spark AR Studio. These enhancements will make it easier to create and place 3D objects in the physical world, such as text, characters, GIFs, stickers, and other items.

Unlocking the possibilities of AR creation is one thing, but you still need to help creators learn how to use your tools.

In its opening speech, Zuckerberg announced a $150 million initiative to train creators in building immersive educational content for people. Meta is therefore expanding its professional curriculum and has partnered with Coursera and edX to make its Spark AR curriculum available to everyone. 

Screenshot, introduction speech, Facebook Connect 2021

To make AR creation possible for beginners with no experience in design, art, or programming, Meta has created a new app, enabled by Spark AR studio: Polar.

Polar app by Spark AR preview

But building a new universe requires instating some ground rules, and one of the main challenges it will bring will be interoperability. There is an important need for governance, and Meta is well aware of it: it needs to get together with other metaverse builders to define standards so everything created in the metaverse is not stock to one place or one platform but can travel throughout this virtual world. To get ahead of this, Meta is already working with game engine developer Unity.