Rumored for years, Apple’s VR/AR device is finally set to make its grand debut at WWDC next June, and it’s got everyone hyped up. But let’s be real, folks; there’s no denying that this is the worst-kept secret in tech. We all knew this was coming, and we’ve been waiting for it for what feels like forever. But the big question is, will it live up to the hype?
I’ve put together this blog to get you up to speed on everything you need to know before the big reveal. We’re going to go deep, and I mean real deep, into all the practical topics you need to be aware of, and we’ll even take a look at Apple’s documentation and product lineup to see what we can learn about their new XR device before it actually launches.
My Love Affair with Apple
With WWDC less than a month away, Apple will finally pull the curtain from their fabled XR device and reveal their plans for entering this new product category. I have been using Apple products as my daily drivers since 2006 when I switched from my trusted Technics sl-1210 mk5s (I still miss you guys) to digital dj’ing with Traktor on my first Macbook Pro. I owned or actively used all products released after that, even the horrible Trashcan Mac Pro and the late 2018 i9 Macbook Pro, yes, the one without an escape key, so my love for Apple must run deep. I have even been at a point in my life where I referred to my MacBook as “my precious.” so stating that I am somewhat of an Apple Fanboy would be the understatement of the year.
However, I’m not completely blinded by my fanboyism. Throughout my career, I’ve spent most of my time working with Windows and Android devices (at some point, I even used a Pixel as my primary phone) and owned all VR devices that came out since the release of the Original Oculus DK1 running some version of Android as well. I know what it’s like to walk on the dark side, so to speak. And while I can appreciate the benefits of these more open platforms, my loyalty to Apple has remained steadfast.
That being said, let’s shift our focus to the topic at hand – Apple’s VR / XR device. As someone who has worked in this space for over a decade, Apple’s entry into the market will be a game-changer. The company has a long history of refining existing technologies to create a superior product. With their vast resources and expertise, it’s hard to imagine they won’t do the same with VR / AR.
While several impressive VR / AR devices have been released in recent years, they have yet to capture the imagination of the mainstream consumer. With Apple’s brand recognition and focus on usability, they have the potential to bring VR/AR to the masses in a way that no other company has been able to do. In fact, Apple’s VR/AR device will be the tipping point that the industry has been waiting for for several years.
Of course, many unknowns exist regarding Apple’s VR/AR device. We have yet to wait for what it will look like, how much it will cost, or what kind of experiences it will offer. But suppose Apple’s past successes are any indication. In that case, it will be a device that is beautifully designed, intuitive to use, and capable of delivering experiences that are truly transformative for users that are more interested in (productivity) apps and tools rather than games.
Where will Apple place the XR device in its product lineup?
With a rumored price tag of 3000 dollars and a clear focus on developers once released, it’s safe to say that Apple will position the device somewhere between the Mac and an iPad if we look at their complete product lineup.
Putting the device here in your product lineup means that, at least for me, expectations will be very high compared to iPad OS. The device should match iPad OS (which in itself is a very capable OS for average use) in terms of functionality or even surpass it to justify the 3000 price tag.
As a developer and content creator, it’s best to approach this device more like a tool to create apps and content than the final mass consumer device. For example, suppose Apple can sell me on the idea that I don’t need to buy a laser beamer nor a 7.4.1 Dolby Atmos speaker setup and get a similar experience with their device; that would justify the 3000 price tag for me as a professional creator mostly doing video work since it will be cheaper than buying the actual setup to create this type of cinema content.
So if it’s a device that will be used less for consuming content and more as a tool to create content for other devices, just like you use a Mac to create content and apps that are being consumed on iPhone, iPad, iWatch, and accessories like Airpods, I don’t expect it to be more capable than OSX in terms of productivity or multi-window use but instead works in tandem with it. For example, it should be the device I review my 3D models in that later will be used in an iPhone app that runs ARkit.
The ecosystem of Apple accessories iCloud, Airplay, Apple ID, and U1?
One of the most overlooked features of the Apple ecosystem (because it’s so smooth, you take it for granted) is that all my devices are linked through Apple ID, Bluetooth, and wifi, and I can switch freely between them without having to reconnect or do any setup. For example, my keyboard and trackpad are connected to multiple devices and move with me from one device to another. The best illustration in use are my AirPods Pro, which constantly changes between my Macbook, iPad, and iPhone throughout the day. If Apple wants me to do serious work on their new device, they must have figured out how to migrate all these accessories to rOS when I put on the headset automatically.
Another super powerful productivity feature for content creators would be Airplay (or even share play) to cast content to a Headset. Currently, I have a couple of devices I can cast a video to, Apple Tv, Mac, and home pods (audio only), so I can’t imagine why they shouldn’t build this in for their new Headset as well; they could even go as far to not only support casting of video or audio but 3D assets as well that already run on the ARkit framework.
Apple has been putting u1 chips in all their mobile devices, accessories, and audio product over the last couple of years, so devices that connect know their relative position to each other. The best example is the spatialized audio functions with AirPods Pro that will lock the audio of the device you are listening to in 3D space relative to your head.
Lastly, they have their airtag products (basically a round u1 chip), allowing you to spatialize essentially any object as long as you put an airtag on it. It is easy to imagine this type of technology will also migrate to an XR headset, where they make much more sense.
What are the new file types I will be dealing with in rOS?
Any good app or game needs amazing assets to populate it. Since we now have to deal with 3D space as well as opposed to an application that is made for flat screens, it’s time to have a look at assets other than images and video that generally populate an app.
USDZ
So, USDZ is a 3D file format developed by Apple and Pixar specifically for Augmented Reality (AR) applications. It’s based on the Universal Scene Description (USD) format, a high-performance 3D graphics interchange format. Think of it like a streamlined version of the traditional 3D file formats we’ve used for years.
USDZ is unique because it allows 3D models and textures to be easily shared between AR / VR platforms. This means that you can create a 3D model in one program and then quickly use it in another without worrying about compatibility issues. And since it’s designed for AR / VR applications, it’s optimized for mobile devices, making it lightweight and quick to load and, therefore, the perfect format for final app distribution.
USDZ supports both static and animated 3D models and can include textures, lighting, and other scene data. It also supports real-time rendering, meaning you can display your 3D models with high fidelity and interactivity on a wide range of devices. This is especially important for AR / VR applications, where users expect to see virtual objects interacting with the real world in real-time.
Overall, USDZ is an exciting development in 3D graphics and VR / AR. It offers a simple, efficient, and versatile way to create and share 3D content across various platforms and devices. Many companies, including Adobe, Sketchfab, and Unity, have adopted USDZ as their primary format for AR content, making it an increasingly popular choice in the industry.
If you want to convert some of your existing 3D models to USDZ, check out this super handy tool from Apple Developers tools.
Dolby Atmos Audio
The great thing about Dolby Atmos is that it is already a standard in high-end video production and cinema distribution. It is supported by video editors like Davinci Resolve and Game engines like Unity through Dolby’s Fmod plugin. Over the last few years, Apple has been slowly integrating Dolby Atmos into all its products and services, like Apple Music and the Apple TV+ video streaming service. If you have AirPods Pro or Max, you can even have it on the go, so Dolby Atmos will be the perfect format for XR because it can render audio in 3D space.
I personally think audio is one of the most overlooked aspects of any XR app or 360 videos and, most of the time, an afterthought because Immersive audio production is currently leaning on B-formats (ambisonic audio formats like TBE and AmbiX) that were not designed to work with XR headsets nor modern DAW’s in mind. Hopefully, when we move to a new standard for immersive audio where the tools are already in place and stress tested, more attention will go to making good quality immersive audio that really adds something to the overall experience.
360, ankered or surface-mapped Video Formats
I have covered almost everything related to immersive video in my company’s knowledge base, so I will keep it short. Apple’s native video player supports a wide range of video formats, but H265 is the way to go as a distribution format since we did not hear anything about AV1 support from Apple. Videos can be downloaded as a complete package for offline playback or streamed, broken up into various qualities as an HLS playlist. And it supports HDR, which will play nicely with those rumored SONY OLEDoS displays in the new XR device.
Apple’s video services and push into live sports broadcasting clearly indicate that Apple is doubling down on video for XR headsets. New, especially for this XR device, is support for 360 videos, videos ankered to a specific point in space, an image or QR code, or a video that can be mapped to a horizontal or vertical plane in 3D space. This will open up many new possibilities for video playback, and a good video player could potentially be a killer app on launch.
For a complete list of supported video formats from the native video player, look at this part of the iOS documentation.
Check out this example from the Gorillaz to see what kind of mixed reality experiences this could power in a headset.
What will be the leading development platform for XR apps?
Since the launch of iOS and later iPad OS, apps have been primarily developed natively with Swift, but since XR apps have more in common with games (at least from a developer standpoint), it’s logical that we also need to look at another development platform other than Apple’s native solution
Swift / ARkit / WebXR
The combination of Swift and ARkit will be a central focus point for any app development on their new XR device, but I still have many questions about how their tools will translate to 3D space and how easy they will be to work with. Obviously, simple 3D apps that can be built only using the ARkit and the WebXr framework will be perfectly doable, but for anything more complex than viewing a 3D model, a game engine will be more suited to the job. Hopefully, we will know more after WWDC what their tool kit looks like and if these tools are robust enough to build more complex 3D applications.
Unity
Unity’s primary use for both iPhone and iPad is mobile games, but this could all change with their new XR device since one of the most significant advantages of using Unity for VR and AR development is its ability to support designing in 3D space and is truly multiple platforms. This means that developers can create applications that can be used on popular VR and AR devices like Meta Quest, HTC Vive, and Pico, as well as mobile devices and tablets. On top of that, about 90% of all VR games are currently developed in Unity, so it makes sense for Apple to target this platform since most XR developers are here.
Unity also offers a wide range of tools and assets that can be used to create realistic environments and interactive experiences directly from their asset store. From 3D models and animations to audio and lighting effects, Unity provides everything designers and developers need to bring their VR and AR applications to life right out of the box.
In addition, Unity’s C# scripting capabilities allow developers to create custom behaviors and interactions for their applications. This means developers can create unique experiences tailored to their users’ needs.
We have been using Unity for years as the backbone of our SDK and all our XR video templates, and it powers both our Link and Operator application, so hopefully, Apple will release some dedicated APIs, Tools, and Plugins to make migration to their new headset a breeze.
Unreal Engine
The Unity game engine’s biggest competitor is powering Fortnite (essentially the one and only true metaverse app up until this point), the Unreal Engine by Epic Games. Currently, Epic is in a legal fistfight with Apple over the 30% percent cut Apple takes on any purchase made on their platform, so it remains to be seen what this will mean for the release and support of new dev tools for Unreal engine when Apple’s XR device will find it’s way to developers.
Arguably the Unreal engine is the more capable game engine of the two in terms of visual fidelity, and since it uses a form of visual programming, it doesn’t rely solely on code to get more complex interactions done. On top of that, they have purchased companies like Sketchfab, Quixel, and Bandcamp, so they have a great library of Assets to build your experience and are well-positioned for VR/AR development.
To get a better insight on Epic’s plan for the Unreal engine, check out this interview with Tim Sweeney, the CEO of Epic, with the Verge, since he is one of the few people who has actually had to say something meaningful about “the metaverse” for lack of a better word.
How do I distribute XR apps?
We became somewhat of an expert in app distribution since we supported thousands of clients get their apps sideloaded on VR devices or helped them publish their apps to one of the VR app stores out there to get their apps distributed. Since sideloading apps to an Apple device is still prohibited under iOS 16 (this might change in iOS 17 for EU users), the only way to distribute rOS apps will probably be going through Apple’s suite of tools to get your apps listed in the App Store or distribute through an Apple Enterprise account. I have documented this process in detail in this blog, but you need three things to start.
- A signed .IPA file (your app)
- An Apple Developer Account or Enterprise Account
- A M-Series Mac
This blog is not the place to go into too much detail about how the exact procedure works for distributing rOS apps. Still, I recommend reading up a little so you know what to expect since it will be more complex than sending somebody an app they can install as we used to do on Android devices. One of the benefits of using Apple’s closed ecosystem for app distribution is that you can also use Apple’s enterprise tools for distributing apps within an organization or use the scholar solution for distribution within learning institutions.
What to expect from rOS?
One of the things that frustrated me the most over the last couple of years using every VR headset out there is their need for a well-thought-out, recognizable, and easy-to-use OS. All headsets out there use a customized version of Android, which is fine when done right. However, having used all VR operating systems in several iterations, I know they vary from frustrating to work with to a downright punishment. Functions like kiosk mode (effectively bypassing the VR controllers and the main menu of the OS) are just solutions to a problem that should not exist in the first place if the OS was designed to be simpler and the controllers easier to use for new-time users.
Luckily Apple is notoriously good at getting the primary menu navigation, gestures, and UI selection right in their first product. Look at iPhone and iPad; yes, it’s still just a grid of apps I swiped through just like 16 years ago or the more exotic app dots UI on watchOS, which was also there from day one. I am still determining what the main UI will look like, but I hope it’s simple and doesn’t require any assistance learning the input method. It’s good to look at something like their newly introduced stage manager UI for Multi App navigation. Although it’s a little cumbersome on OSX and iPadOS, it can work well in 3d space.
Apple has been working for quite some time now to get OSX to look more like iOS and vice versa. Take the settings menu, for example, it’s almost identical on all platforms, which makes switching between these devices much easier, and you don’t have a learning curve when buying a new device. So don’t be surprised if the rOS looks a lot like we already know from their other Apple platforms; a little bit boring, maybe, but at least I don’t have to find the same settings menu over and over again because it’s been placed somewhere differently with each OS update (yes I am looking at you, Meta!)
How to interact with reality OS ?
For a truly intuitive way of interacting with rOS (without the help of a keyboard or trackpad), we have to ask ourselves a couple of simple questions, How do I select an item or button? How do I click? How do I get back to my main menu? And how do I scroll or swipe? Although these questions may sound simple, all current VR headsets have no answer to this that works without setup, explanation, and much trial and error with complicated VR controllers. I remember the first time I gave an iPad to my 90-year-old grandmother, and the interface was entirely natural for her. I expect This level of polish and simplicity from Apple for their XR device.
Nobody has figured this out yet because it’s damn hard, and a lot of hardware and software need to work together to make this possible. Let’s start with selecting a menu item; this needs to be done with eye tracking since it is the only accurate way that doesn’t strain my body over time. We have seen this first exemplary implementation in the PSVR2, and it really works like a charm.
Secondly, let’s look at gestures like, Clicking, Scrolling and swiping, and going back to the main menu. Although I don’t think hand gestures a la minority report will be the primary input method since it won’t be comfortable for extended periods, we will see some hand input. The best device to look out for is the iWatch since it already has some of these functions built into its accessibility functions called assistive touch. Although now branded as accessibility functions, I won’t be surprised if they repurpose these gestures for their XR device, where they make a lot of sense since you don’t need to lift your hands, and you could even use an iWatch to register these inputs. On top of that, you have some form of haptics, and you could even use the digital crown for scrolling.
Passthrough
One of the biggest problems with current VR headsets is the complete isolation once you are in there; most high-end VR headsets have a form of passthrough, but the camera quality is so bad it’s unusable without getting VR sickness. Since Apple wants to mark its headset as a Mixed Reality device rather than a VR device, passthrough will probably be the default option, and you switch to full VR if the application or game needs it. Essentially they already made an AR device only without the transparent screens (since they don’t exist yet), which makes sense from a strategic standpoint, and for developers who will be committing for the long term to this platform.
How you control passthrough is another thing that could be borrowed from the Apple Watch is its digital crown (something they also did with the Airpods Max) to switch between full VR, and the passthrough camera’s to let the outside world in. If you have an Apple Watch Ultra and the Wayfinder watch face, you probably be familiar with how it can switch from day to night mode. Think something similar, only now you are switching between full VR and mixed reality just by turning the digital crown on your headset. They could even use the digital crown on your watch, for that matter, since it is ergonomically better positioned than the digital crown on your headset.
Seamless Login and eye calibration.
Many rumors have centered around hardware leaks over the last few years, but personally, I am more interested in how these hardware components will work together in one seamless experience; simply put, it just needs to work, which is definitely not the case with the current generation of VR headsets.
Almost all standalone VR headsets use an XR2 chip by Qualcomm. Although not a bad chip by any means, it’s only capable of doing a few tasks at a time without getting too hot, and it’s definitely not capable of processing a lot of sensor input, let alone while running multiple apps. It would be exciting to see how Apple’s M2-powered chip line will meet the task since, in my opinion, it was Apple’s most prominent break trough from the last couple of years.
Suppose we can take anything from OSX, iOS, and iPad OS. In that case, they all use a form of biometric device unlock (TouchID, FaceID, or wrist detection), and after that, your device is ready to use; however, with VR headsets, this becomes a lot more difficult.
A couple of years back, Apple bought SensoMotoric Instruments, which by that time were industry leaders on anything related to eye tracking and pioneered eye tracking in VR headsets very early on. Hence, it’s safe to assume their core tech migrated to the Apple XR headsets very early in development and got perfected over time. Iris scanning seems the way to go for unlocking the device and eye input in rOS after that; to work smoothly, unlocking the device could also be the point to do some much-needed eye calibration.
Body , Face, and Room mapping
Although they never released a VR device, Apple has been gathering data and perfecting its sensor and face / room mapping tech by outfitting each iPhone with ToF sensors for years. Now let’s look closer at the sensors at play for mapping the room and your body. They will be similar to the ToF sensors currently used for face ID at the front of an iPhone and AR apps and the camera with the ToF at the back, only with more precision and running in real-time on its own custom signal processing chip. And it needs its own dedicated processing chip because you will need at least a dozen ToF sensors working in real-time to track your body, face, and environment. Over time and later iterations of the headset, you could probably do it with fewer sensors levering some AI magic.
Where to look for inspiration on UI?
I have been using my 12,9 inch iPad Pro as my primary mobile device for the last year to force myself to use it to execute more complex tasks other than just media consumption and web browsing since I believe there will be a lot of similarities between rOS and iPadOS in terms of how you interact with the Operating system and what apps it will launch with.
With the latest rumors from Bloomberg regarding app compatibility with iPad apps, it is obvious to look for inspiration over at iPad OS since the rOS store will probably be populated with iPad apps at first. Working on the iPad Pro 12,9 inch with the latest m-series chip will indicate the new device’s capabilities and what to expect from the OS and its apps regarding features and performance.
I have compiled a list of some apps that you can check out since I can see them easily translate the UI to 3D space and rOS, or I expect them to be there at launch since they will be an integrated part of rOS.
Apple’s Native Apps:
this app works great on iPad but could be much more powerful in VR
this will probably be the best place to look for how your digital avatar will look before we go to full-body scanned avatars or metahumans
An excellent place to look at how app switching could work in 3D space and how files could move from one app to another.
Apple TV+, Apple Music, Apple Podcast app
All devices have the same design language, so I don’t expect them to differ on their XR device.
Reality Scan + Reality Composer
Reality Scan is an app created by Epic Games that allows for capturing real-life 3d models with an iPhone using photogrammetry. Still, Apple will put this kind of tech in their headset or the camera app. Reality Composer is how you get these 3D scans into ARkit.
Third-Party:
It’s one of the few games I play, but I can’t help but wonder how this will look in VR; it’s almost made for it, even better than for a flat screen.
Great app for checking out how real-time spatialised audio works while using AirPods
A meditation app that with a UI that could easily work within 3D space as well
Final thoughts
So there you have it; these were my takes on the rumored XR Apple headset; hopefully, I got you as excited as I am for these new generations of headsets and what content creators and developers can do with it once it get into our hands.