Technical Overview of Microsoft Mesh
Mesh is Microsoft’s vision for the future of mixed reality. Mesh is a team platform that allows you to have shared virtual experiences on a variety of devices which was launched on March 2, 2021, during the Microsoft Ignite 2021 event.
Why is there a need for Mixed reality?
Mixed Reality is a blend of physical and digital worlds, unlocking the links between human, computer, and environment interaction. In simple terms, it allows users to create a new environment where there is co-existence of both digital and physical objects.
The applications of MR are growing widely day by day from being applied to remotely provide support to construction workers to act as a tool for virtual testing for engineers. One of the popular examples that proves why we need MR is in Healthcare. In this sector, MR can be used by doctors to practice surgeries and gain better data visuals without any harm or risk to human life. Another example of MR is acting as a key tool for rendering physical objects of the real-world in the form of 3d holograms thereby reducing the cost of resource acquirement and utilization.
MR is needed mainly in the manufacturing sector where it allows users to view types of equipment in their digital version by pointing their devices accordingly.
Technical training can also be provided by using holoportation supporting devices or headsets. Thereby MR is needed to provide training and manufacturing solutions at reduced repair times and low-cost solutions.
To brief it up, MR bridges the gap between AR and VR. Its unique concept pushes the industries for the betterment of their solutions in all dimensions.
What are the underlying problems while creating mixed reality and the role of Mesh in it?
Some underlying hard problems that prevent developers from creating these immersive experiences are notably:
- A lot of time and resources are needed to represent users in MR with pertinent realism.
- A non-trivial problem is to keep a stable hologram in a shared MR space.
- Bringing hi-fi 3D models to support file formats is a pretty tough task.
- A complex job in geographically distributed MR sessions is to synchronize actions and expression.
These are some challenges developers face for enabling MR experiences which Microsoft Mesh aims to unravel.
The biggest application of Mesh is in Education. This tech furthers our ability to demonstrate conceptual models in STEM far better to casual audiences. Bridging that scientific literacy gap is crucial nowadays.
Furthermore, while learning medicine advanced visuospatial perception is often required—be it for assessing organ placement during surgery, or understanding physiology at the cellular level.
To quote Langston, “Engineering or medical students learning about electric car engines or human anatomy could gather as avatars around a holographic model and remove parts of the engine or peel back muscles to see what’s underneath. Colleagues could simply get together and chat in a shared virtual space, or companies could use Microsoft Mesh-enabled apps to offer virtual all-hands meetings or training to employees around the world.”
Excluding the technical jargon, What actually is Microsoft Mesh?
“You can actually feel like you’re in the same place with someone sharing content, or you can teleport from different mixed reality devices and be present with people even when you’re not physically together,” explains Microsoft’s Alex Kipman, the inventor of Kinect and HoloLens.
If you have been waiting for a science fiction tech or mixed reality product like Tony Stark has, then Mesh is the one you need to keep your eyes on.
According to Microsoft, Holoportation is a technique that allows users to appear as avatars of themselves in space. Microsoft Mesh is said to support holoportation. Though holoportation allows only the creation of avatars, Microsoft promises communication between people in their photorealistic forms.
Mesh has been regarded as the greatest and most innovative product in the field of AR and VR. Let’s check out the core components of Microsoft Mesh.
- HoloLens2: With its immersive mixed reality capabilities, HoloLens 2 and its Mesh App can connect and co-create with others from the comfort of wherever you’re working.
- VR Headsets: Using VR Headsets, you can meet, share, and work with other users in virtual spaces with Mesh.
- PCs and Mobiles: You can join Mesh-powered meetings and design sessions using your very own PCs and smartphones.
- Mesh-enabled apps: On top of the development platform, Microsoft Mesh also delivers some app experiences that bring the platform alive.
Mesh is enabled by a comprehensive platform and a collection of tools. The heart of this developer platform is Azure.
The HoloLens 2 Mesh app and AltspaceVR are instantiations of the collaborative experience Mesh can light up for immersive headsets.
As a developer, you don’t need to worry about core infrastructure around billing, audio/video transmission, and the underlying live-state management capabilities because of the presence of services like Azure Active Directory and Microsoft Graph. The following are the capabilities of Microsoft Mesh:
- Immersive presence: The Mesh platform, with its AI-powered motion models to capture motion and expressions, comes with an avatar rig and a customization studio, so you can use the out-of-the-box avatars that represent participants depending on devices they’re joining from. It also enables the most photorealistic 360o holoportation with outside-in sensors for life-like representations.
- Spatial maps: This framework enables content to be anchored, device point-of-views to be shared, and 3D models to be collaborated on thereby enabling Mesh to have a global understanding of the space/environment the user is in.
- Holographic rendering: This gives the flexibility to design apps that can optimize for latency vs fidelity depending on the device and natively rendering 3D file formats in Mesh-enabled apps.
- Multi-user sync: Creating a common perspective of the hologram and each other within a collaborative session is augmented with spatial audio in Mesh that creates a sense of being in the same physical space in a multi-user scenario.
A cross-platform developer SDK by Mesh to leverage the capabilities and core platform feature thereby providing the developers the freedom to work on platforms of their choice like Unity alongside native C++ and C#, and upcoming support for Unreal, Babylon, and React Native.
This product is built on Microsoft Azure, enabling developers to build immersive, multi-user, cross-platform mixed reality apps.
Mixed Reality is going mainstream across consumers and commercial, liberating screen-bound experiences into instinctual interactions in your space, among your things, with your people. Microsoft Mesh feels like a science fiction movie come to life with its mind-blowing design. While a lot many product-based companies are working on innovations MR solutions, Mesh has proven to be the next big thing.
Mesh uses AltSpace VR, a social network that came out back in 2017, to help create virtual avatars for users. Mesh will soon be integrated with Teams thereby providing a far more futuristic call than any other team meeting platform.
To experience the Mesh MR, all you have to do is wear a HOLOLENS 2 headset, enter a virtual meeting, and you’re in for the ride!
All the participants in the virtual meeting can take up a virtual avatar, and it would feel like you all are in the same room together. The pandemic has resulted in enforcing tech innovations to greater heights and mesh was one of them. Microsoft is also making Mesh available on a variety of devices, including the HoloLens 2, most virtual reality headsets, tablets, smartphones, and PCs.
Powered by the computing and AI capabilities of Azure, Mesh is also designed to allow architects, designers, and engineers that handle 3D modeling and design to work together across the globe.
Despite the advancement, Mesh has its own set of difficulties like the cost of VR headsets, HOLOLENS 2.Another thing is that actions like jumping from a 2D screen are possible, the experience isn’t immersive enough in the headset.
From having a virtual MR meeting on Teams to a virtual concert to an MR gaming experience, the possibilities are limitless. But the reality and success of Mesh depend on the way Microsoft markets its technology and overcome the limitations mesh currently faces.