Augmenting the TV experience

In recent years, we have seen an increasing trend in the use of augmented reality (AR) technology in various fields, including entertainment. One of the latest trends in the entertainment industry is augmenting the TV experience. The AR technology has the potential to enhance the viewing experience of users and provide a new dimension to their entertainment experience.

Firstly, let’s look at the benefits of having a synchronized AR experience with TV. By combining the two technologies, users can have an immersive experience that goes beyond the boundaries of traditional TV viewing. AR technology can bring characters and objects to life and place them in the user’s surroundings. This provides a more interactive and engaging experience for viewers, especially for younger ones who love to see their favorite characters come to life.

In addition, augmenting the TV experience can also provide users with additional information and context about the TV program they are watching. For instance, AR overlays can give users information about the characters, settings, and even historical facts related to the program. This can add another layer of engagement and educational value to the viewing experience.

“For All Mankind: Time Capsule” is the companion AR experience of the eponym Apple TV+ show. It allows you to access exclusive content and dive deeper into the story.

However, synchronizing AR experiences with TV streams also presents a set of challenges. One of them is the synchronization itself. Another challenge is, during a live show, to manage the delay in transmission between the TV program and the AR experience, which can result in a lag or desynchronization. That would ruin the experience for users.

Another challenge is the development of AR content that is compatible with different TV programs. AR experiences need to be carefully designed to ensure they are suitable for the content of the program and do not distract from the narrative or disrupt the viewing experience.

Despite these challenges, there are several ways to synchronize AR experiences with TV streams. When using a second-screen device to launch the AR experience, such as smartphones or XR glasses, it is triggered by specific moments or cues in the TV program, such as a particular scene or audio cue. The second-screen device can then display the corresponding AR overlay, synchronized with the program.

Image recognition is another method for augmenting the TV experience. In this approach, the AR app uses image recognition technology to detect specific scenes or objects in the TV program. Once detected, the app can trigger the corresponding AR overlay, providing a synchronized experience for the user. This method requires a high level of accuracy and precision in image recognition technology to ensure the AR experience is triggered at the right time and place.

“Remembering: the AR experience” is the AR companion experience created by Disney+ for its eponym short film. It invites you to step into the universe of the movie.

Synchronized AR experiences with TV programs have the potential to revolutionize the way we watch and engage with entertainment. But, there are some challenges associated with using watermarking or image recognition for synchronizing AR experiences with TV streams.

Watermarking requires modification of the audio before airing, which means that the synchronization has to be done in advance, and it can be a challenging and time-consuming process. Additionally, the watermark has to be embedded in a way that is imperceptible to the human ear, which requires sophisticated technology and a careful balance between the watermark and the original audio quality.

On the other hand, image recognition can also present some difficulties, particularly in terms of accuracy and performance. The AR app needs to have a high degree of accuracy in detecting specific scenes or objects in the TV program, which can be difficult in complex scenes or in low-light conditions. Additionally, the performance of the app may be affected by the processing power and memory of the device, which can limit the speed and accuracy of image recognition.

Both watermarking and image recognition methods also require a robust and reliable network connection to ensure real-time synchronization between the TV stream and the AR experience. This can be a challenge in areas with poor network coverage or slow internet speeds, which may result in lag or desynchronization between the two technologies.

Overall, while watermarking and image recognition have the potential to provide a seamless and immersive experience for users, they also require careful planning and execution to ensure a high level of accuracy and synchronization.

Maybe a better solution lies somewhere…

At Immersiv, we came up with another approach. In order to create seamless companion AR experiences with TV, we use audio recognition. The principle is simple: your device picks up the audio from the TV program, then our app analyzes it and is able to pick up where you are in the program and synchronize the AR experience accordingly. 

And thanks to the infrastructure we’ve deployed, we are able to do so on live TV. A great use case is our work with Deutsche Telekom during the 2022 FIFA World Cup. Throughout the competition, whenever a game was airing, fans could display an AR mini-pitch on their coffee table, and while the game was playing on their TV screen, the action on their mini-pitch was synchronized with what was happening in the stadium! They were also able to display all kinds of live information on their mini-pitch like players’ or teams’ stats, heatmaps, live speed… All of which was synchronized thanks to audio recognition.

“Magenta TV FIFA WM 2022 AR” is a companion augmented reality TV experience we developed for Deutsche Telekom during the 2022 FIFA World Cup. It gave fans another perspective on the game by recreating the action on their coffee table.

Augmenting the TV experience provides an immersive and engaging entertainment time for viewers. However, the technology presents challenges in terms of synchronization, content development, and network connectivity. While watermarking and image recognition are common methods used to synchronize AR experiences with TV streams, they require careful planning and execution to ensure accuracy and synchronization. 

At Immersiv, we have developed an innovative approach using audio recognition to create seamless companion AR experiences with TV programs. Our product ARISE provides users with an interactive and engaging experience that is synchronized with the program. With this technology, we have successfully deployed synchronized AR experiences during live TV broadcasts, enhancing the viewing experience for fans. Synchronized AR experiences with TV programs have the potential to revolutionize the way we watch and engage with entertainment, and we are excited to continue exploring this technology and its possibilities.