Facebook announces the Passthrough API, which allows Oculus Quest 2 developers to use real-world images collected via sensors to create mixed reality apps
Developers will get the experimental feature with Oculus' next SDK — Facebook announced a new API that will let developers incorporate video …
The VergeMitchell Clark
Context & Ripple Effects
Oculus Quest’s inside-out tracking supplied the sensor foundation before Facebook opened it to application makers. The Passthrough API turns that tracking and spatial-sensing stack into an experimental developer input for mixed-reality experiences.
The move also follows App Lab’s alternative distribution path for Quest developers and precedes the broader Presence Platform SDK suite. Later Quest 3/3S coverage shows Meta continued treating camera access as experimental and separately constrained app-store distribution.
First-order effects
Quest 2 developers gain experimental access to real-world imagery from headset sensors, allowing mixed-reality apps to combine a user’s surroundings with virtual content.
Facebook expands the Quest SDK from headset interaction and tracking capabilities toward applications that depend on live environmental perception.
Second-order effects
Developers using App Lab have a route to test and distribute mixed-reality software outside the main Oculus Store, increasing the practical value of the new API for smaller Quest publishers.
Facebook’s subsequent Presence Platform SDKs package related mixed-reality capabilities more broadly, raising the incentive for developers to build around its platform rather than isolated headset features.
Third-order effects
The later experimental Quest 3/3S camera API indicates that access to a headset’s view of the physical world is becoming a continuing platform layer, while store-policy controls remain a gate on commercialization.
Mixed-reality software ecosystems are likely to be shaped not only by headset sensors but by the platform owner’s API permissions and distribution rules.
The trend: VR headset makers are evolving their tracking systems into controlled perception APIs for mixed-reality developers.
Developers will be able to start building and testing mixed reality experiences on Quest 2 with Passthrough API Experimental—an important step in unlocking the next generation of immersive experiences: https://ocul.us/... (going to miss getting this FAQ at my AMAs =)) https://twi…
@prvncher
@prvncher on x
Pretty wild to see Quest becoming a dev kit for the future of mixed reality applications. Still feels like we're stuck in the CRT days, but the fundamental pieces are all here! https://twitter.com/...
@dilmerv
Dilmer Valecillos on x
Ok this is a huge win towards #MixedReality 🥽 🔥 ! Having a device that does VR and allowing mixed reality is sooo huge, this is a $299 device and there is nothing available today to learn with such a low cost of entry. https://twitter.com/...
@casandchary
@casandchary on x
Wow! Can't wait to see what devs come up with. Hopefully this also means we get to record it too as passthrough is currently blocked from being captured. https://twitter.com/...
@ali_heston
Alexandria Heston on x
@kentbye Agreed - I think the ideal use case for Oculus with this is for a softer transition of their VR devs into their eventual AR product.
@austen
@austen on x
Excited for 10 years in the future when everyone is questioning the SEC and DOJ on how they allowed Facebook become a monopoly in the obviously insanely valuable virtual reality space (when in reality no one really cared)
@ddiakopoulos
Dimitri Diakopoulos on x
A bit disappointed that the Oculus Passthrough API doesn't provide images for fun computer vision experiments, but definitely the right decision from a privacy point of view https://developer.oculus.com/ ...
@kentbye
Kent Bye VoicesOfVR on x
@mattmiesnieks I've seen higher resolution and low-latency VR headsets with passthrough, but the inevitable offset between my actual eyes & the virtual eyes/camera threw off my whole proprioceptive orientation & body schema. I'm sure I'd adapt, but VR passthrough has a weird disc…
@fabio914
Fabio Dela Antonio on x
It'd be really cool if Oculus could also provide a “person segmentation” feature similar to that of #ARKit. Imagine being able to see real people around you transposed into your VR scene, multiplayer VR with people in the same room would look really cool! https://twitter.com/...
@robertjala
Robert Aldridge on x
Pretty frustrated that they are neglecting Unreal here. Medieval Designer could have been the first to implement the mixed reality capabilities of the quest because it is 100% perfectly built for it, but because of nonsense decisions like “unity first” it won't be. https://twitte…
@sadlyitsbradley
Brad Lynch on x
Good stepping stone for them to get developers into the full on AR pipeline and show users the potential But Quest 2 Passthrough looks way too garbage for it to be used as a dedicated AR/MR device https://twitter.com/...
@uploadvr
@uploadvr on x
Facebook shared some details about its experimental Passthrough API to enable new kinds of mixed reality apps for Oculus Quest 2. https://uploadvr.com/... https://twitter.com/...
@mattmiesnieks
Matt Miesnieks on x
@kentbye I think that's the right way to think about passthrough video on a VR HMD form factor. A “feature” to make VR a bit more convenient, and a great way to prototype and test various ideas for future outside-wearable see-through AR glasses
@kentbye
Kent Bye VoicesOfVR on x
There's going to be a lot of really interesting AR, Mixed Reality, & IoT apps that are prototyped using a VR headset with this new Passthrough API. Rather than thinking about VR vs AR markets, I prefer to think of VR helping bootstrap the HCI, UX, & UI for AR apps. Each has +/- h…
@rengle820
Ryan Engle on x
Wow! Better than I was expecting. Now we just need some high quality cameras! https://twitter.com/...
@vibronet
Vittorio on x
This is fantastic. By making true AR API available on cheap dev devices today, it will jumpstart innovation and the development of apps that will make future AR glasses awesome from day 1 https://twitter.com/...
@tovanbo
Thomas Van Bouwel on x
This looks amazing! Can't wait to try this in Cubism 👌 https://twitter.com/...
@jasteinerman
Jake Steinerman on x
The Oculus Quest was never just a VR headset - its been an XR headset all along. Now more obvious than ever with the new Passthrough API https://twitter.com/...
@dennyskuhnert
Dennys Kuhnert on x
Cannot wait to experiment with this! 👀 https://twitter.com/...
@mattmiesnieks
Matt Miesnieks on x
@kentbye The passthrough tech will get awesome & close enough UX to see-through.... But the real issue why passthrough won't ever hit scale for AR use-cases is it is a solution to get around the huge black box on your face, that no one would ever wear in public :)
@purplemikey
Mikaël Dufresne on x
@boztank excited to see what devs will manage to do with passthrough :) Very exciting.
@dens
@dens on x
Wow this is pretty slick (also never thought of Oculus as an AR device) https://twitter.com/...
@_jasoncooper
Jason Cooper on x
🤓 @Oculus just enabled the ability to create and test #MixedReality experiences on Quest 2 with their Passthrough API. This is now possible on a $299 #VR headset. Pretty remarkable. https://twitter.com/...
@david_jagneaux
David Jagneaux on x
This is so cool! https://twitter.com/...
@hmltn
Ian Hamilton on x
Quest 2 is now a $299 AR development kit. https://twitter.com/...
@hrafntho
Hrafn Thorisson on x
Oculus announcing that v31 update for Quest will have passthrough API 🔥 It'll be an exciting new chapter when headsets get good color passthrough. Still a solid range of practical and interesting applications possible even with current black and white.