In our previous blog we defined our Cloud encoding profiles, In this blog we will go in-depth on why we made our choices regarding video quality and video compression and what to keep in mind when setting up your own encoding profiles. The industry really professionalized over the last couple of years and we are slowly seeing the adoption of industry-wide standards which can be targeted both with hardware and software improvements. Camera technology dramatically improved so the need for higher resolution really started to become an issue while the headset market became much less fragmented and easier to develop for. On top of that, the headset hardware improved dramatically which enables higher resolutions in the first place.
After several iterations, all the developer/publishing tools to get your apps published to the storefronts are finally up to industry-standard which essentially created an audience for us to target with these higher resolution videos.
Global CDN
Headjack distributes videos using the optimized global CDN Amazon CloudFront and in the last couple of years, we’ve handled hundreds of thousands of up- and downloads from around the globe for a variety of VR studios. After numerous iterations and rewrites of our upload and download protocols, it’s now the most stable upload solution out there for handling huge video files. When paired with a fiber-optic internet connection upload speeds exceed 1Gbit/s making it possible to upload even the largest master videos to our server in no time.
We offer custom CDN set-up for enterprise clients if you require any other CDN solution due to security protocols. We also build custom APIs for direct communication with our platform, so you can use your own server infrastructure or web front-end.
Why H.265 and H.264?
Let’s start with H.264 because it seems like this format just won’t go away. Although it used to have strict limitations in regards to a 4096px resolution limit, hardware support has improved massively, enabling up to 8K resolution in the H.264 codec in theory. Although resolutions up to 5760×2880 work perfectly well for some mobile VR headsets like the Oculus Quest (even being recommended), this is certainly not the case for PC-based headsets. They usually don’t have support for GPU acceleration for anything above 4096px on H.264 and therefore will process the video data on the CPU which practically makes anything above 4K unplayable.
So to enable higher resolution for PC-based headsets and reduce file size in these higher resolutions it’s recommended to use the H.265 codec. Check the table below with all of our tested resolutions and recommended codecs for each platform.
Optimized for Post-Production Pipelines up to 5.7K
To better align with VR production pipelines we targeted a 5.7K baseline for all of our monoscopic video playback in VR headsets. 5.7K seems to be the go-to standard for the coming years before we move over to 8K resolutions after that. Currently, 8K production pipelines are very expensive to set-up and operate, most production software is not yet ready to do the job. Almost every computer, render farm and network will have their occasional meltdown moving all of those pixels through the pipeline. Learn more about ours here. Most of the hardware and software makers are now targeting 5.7K since it seems to offer the best trade-off in terms of headset display resolution vs. app playback performance. John Carmack at Oculus has also noted that future mobile hardware roadmaps seem to indicate growing support for 5.7K resolution specifically, likely supporting 60FPS at that resolution or even the 5.7K x 5.7K stereoscopic format eventually.
Fixed Stereo resolutions
Another big change in terms of resolution is that we are now using formats that are much more common and practical in VR production for stereoscopic footage, especially 360°. Due to codec limitations, we had to cram as many pixels in a UHD 16×9 video container for stereo footage which caused a lot of resolution loss. Since hardware has gotten better over time we are now able to play dedicated 1:1 stereo formats for all headsets maxing out at 4096×4096 (30fps) for mobile VR headsets and 8192×8192 on high-end PCs.
180° Video and Custom Formats
Headjack has support for a variety of other immersive video formats like 180° equirectangular video, both monoscopic and top-bottom or (recommended) side-by-side stereoscopic. It’s even possible to set a custom horizontal and vertical field of view (FOV) if you are using exotic camera systems or post-production workflows. For mono 180° video it’s recommended to use the 1:1 aspect ratio compression formats (see fixed stereo resolutions above) and use a 2:1 format for side-by-side stereo 180° footage.
Unity Overhead
Building on a game engine as we did with Headjack has both its pros and cons. Probably the biggest advantage compared to a native video player app is that you are building on a very solid development platform fine-tuned over the last 15 years to give you all the developer tools you need to build a fully-featured and content-rich VR application. On the other hand, all of this came at a price in terms of playback performance and heat management due to the engine’s overhead. Additionally, video playback in Unity has long been seen as the inferior technology compared to native video playback but the software caught up over the last couple of years, we can make use of a variety of players all of them better than the standard Unity video player they are delivering with their engine. As of now, there is no difference in what video resolution you can playback in a Unity-based app compared to a native app for both mobile and PC-based VR headsets, making game engines the obvious choice for VR video playback.
Video Player SDKs
Headjack uses the highest performing video playback technologies on each platform we support: ExoPlayer on Android devices, Windows Media Foundation on PC-based headsets and AVFoundation framework on iOS devices. Together with our targeted encoding profiles, you should always get the maximum video resolution out of your VR device with the least amount of friction. Make sure you always have the right codec package installed for Windows to enable H.265 video. Microsoft had to remove native H.265 video support from Windows 10 due to licensing issues. Fortunately, they have provided the H.265 codec as a free download from the Microsoft Store.
CPU Levels & Heat Management
The biggest cause of app crashes and poor video playback on mobile VR hardware has to do with CPU level and heat management. All Headjack apps are locked at a specific CPU level guaranteeing enough power for smooth playback on mobile VR Headsets. And without suffering from render artifacts that are present if you switch between different CPU levels during playback, which is, unfortunately, the default behavior of some VR devices.
TBE and Spatial Audio
Another contributor to exceeding CPU limits is the use of TBE as a spatial audio format The (much less used) alternative is 2nd order AmbiX (9 spatial channels + 2 head-locked stereo channels) served as a separate audio track sometimes has a dramatic impact on CPU levels of mobile devices. the standard format now is 2nd order TBE (8 spatial channels + 2 head-locked stereo channels) packaged as a 10 channel Opus audio track in a .MKV video container, but since we are not completely satisfied with .MKV playback in our video players delivering the spatial audio mix as a separate TBE is the only workable solution we have right now.
Refresh Rate Headsets
In order for smooth playback, all of our apps run at 60 FPS, 90FPS or 120FPS. For Oculus Quest which normally runs on 72FPS, we made the choice to lock the app at 60FPS in order to minimize video playback artifacts and synchronize better with 30FPS video.
Fixed Bitrate vs Constant Quality
In the past, we always opted for fixed bitrate encoding because spikes in bitrate could jam even the most beefed up VR PCs out there and can cause significant issues when streaming video. A fixed bitrate has one major drawback and that is of course file size. That is why we opted for a constant quality (CRF) compression algorithm for all of our downloadable cloud encodes to preserve maximum image quality in relation to file size. Read everything about CRF here and it’s advantages here. Currently, we encode at a CRF level of 18 with H.264 video and a CRF level of 17 with H.265 video, which strikes the best balance between preserving image quality in VR and file size.
8K tiled Streaming
Tiled streaming in any shape or form is undoubtedly the way forward in the coming years to push resolution beyond the 5.7K mark for mobile devices and make high resolution (live) streams available. Of course, we will first have to wait for mainstream VR hardware capable of displaying 8K to arrive but we know both the Khronos Group, Fraunhofer Institute, and numerous start-ups like Tiledmedia and Visbit are working on tiled streaming so we will see what will be the dominant technology when the time comes. The biggest bottleneck at the moment is still the (post-)production pipeline. Luckily our camera issues are a thing from the past with the availability of cameras like the impressive Insta360 Titan.
Unfortunately, most post-production pipelines are not remotely equipped to handle vast amounts of unprocessed 8K video files, let alone 11K. You’ll have to spend tens of thousands of dollars to get your pipeline and hardware up to speed to handle even the simplest editing tasks. Minor compositing like rotoscoping or some simple rig removal will be insanely expensive and slow. The last difficulty is that most common editing programs like Premiere and After Effects barely support a dedicated 8K workflow as of now. So you’ll have to resort to more expensive and complicated solutions like Nuke to get a stable workflow.
In the end, most video compression is done based on personal preference and experience from the editor who is doing the final export? With the latest Headjack updated we opened the door for artists and developers to work more closely together and get the maximum out of their hardware. Since cloud video compression is not our main business model we don’t have to force you into a proprietary video format to justify the costs of your monthly subscription and you can decide what video format to use for final distribution.
If you need help setting up a custom VR video pipeline or you want to get other SDK integrated into your apps than we normally use, please reach out to our support team.
