Apple has major plans for augmented reality (AR). These started with ARKit, a new augmented reality platform launched at its annual Worldwide Developer Conference this year, and then built into iOS 11. That means you can try out AR apps for yourself right now if you own an iPhone X or any other iOS 11-compatible device, but Apple intends to do something far greater with the emerging technology than simply facilitating a few apps.
“AR can be really great,” Apple CEO Tim Cook said when asked about Pokémon Go during an earnings call in 2016. “We have been and continue to invest a lot in this. We are high on AR for the long run, we think there’s great things for customers and a great commercial opportunity.”
Apple AR glasses, or an augmented reality headset, may be part of this, according to a growing number of rumors. Leaked by three alleged Foxconn employees, it’s apparently known internally as Project Mirrorshades, but this is unlikely to be its release name. Exactly what form it will take is also unknown, but the latest rumors suggest that Apple is looking to change how we think of both AR and VR headsets.
Here’s everything we think we know about it, and Apple’s AR master plan, so far.
When is it coming and how much will it cost?
Apple’s augmented reality device is still in the early stages of development. Famed Apple analyst Ming-Chi Kuo predicts we will see Apple’s AR glasses in 2020. This prediction roughly lines up with an earlier rumor from Bloomberg that stated Apple would introduce its AR in 2019 and begin shipping the following year. Apple partner Quanta has also stated publicly that it is working on an AR project for a major client that’s due in 2019.
What will Apple’s AR headset be like?
Apple makes beautiful hardware, so we shouldn’t expect any augmented reality product to be an ugly mess of parts and cables — but what will it look like? According to a now-public patent (found by Patently Apple), Apple is experimenting with a headset design that matches earlier rumors. While it’s not clear whether the headset would be specifically intended for VR or AR use (the patent mentions both), it’s clear that Apple believes it is important to reduce the size and bulk of such a device, and is experimenting with designs to that end.
For example, this patent uses multiple lenses to achieve the same effect as larger headsets, arranging them into what is known as the “catadioptric optical system.” More commonly seen in telescopes, this arrangement is a compact way to focus light, and helps to eliminate the colors that can sometimes be seen on the edges of your vision in VR or AR — or “chromatic aberration.” The smaller arrangement would help to reduce the size of the headset, which would be an important step toward making AR glasses usable in real terms.
While there was no mention of audio in this particular patent, it’s fair to assume earlier leaks will likely pan out, and that Apple is working on a system involving DC motors that produce sound by vibrating through the small bones of the user’s ears, like Google Glass.
The patent also mentions of gaze-tracking in the headset, as well as the ability to head-track in 3D space without the use of external sensors — which would be a huge win for Apple if it also succeeded in shrinking the size. As for controls, the patent mentions a wide variety of controls, and it’s probable that Apple hasn’t fully decided on a control method yet, mentioning everything from microphones, to keyboards, to touchpads. It’s likely that controlling Apple’s AR glasses may also be similar to Google Glass, and previous rumors held that at least one prototype used a touch-sensitive strip on the arm to answer calls and control the volume, along with gestures to navigate through menus. Those rumors also detailed a combination magnetometer and light sensor detects when a wearer shakes his or her head
]and responds accordingly — a Tinder app user, for example, could shake their head for no or nod for yes. That seems likely, based on this most recent patent.
This is Apple we’re talking about, so any final AR device will need to be suitable for everyone, with the patent going to great lengths to discuss the importance of a smaller headset so as to reduce fatigue. Moving away from the patent, previous rumors spoke of two sizes for the headset or glasses, and at least three colors, with the potential of different colors being added seasonally. The shape is also rumored to be based on a classic, round “P3” frame. Of course, the design is likely to change over time as Apple continues to refine.
Corning, best known for its Gorilla Glass product used to cover smartphone screens, is said to have been working on specialized glass — specifically for AR glasses and headsets, Variety reports. Corning has been an Apple supplier since the first iPhone in 2007, and in 2017 Apple invested $200 million in the firm, specifically for research and development.
Another longtime Apple partner, Quanta, which among other things builds the Apple Watch Series 3, has said it’s working on an AR project for a client suspected to be Apple. The company described the product as “headset-like gadget with a fully transparent lens that allows users to see through and interact with the environment.”
A recent report in Bloomberg states that Apple will develop its own chip for the headset instead of using an existing chip. The same report says the company will create a new operating system for the device, much like it did for the Apple Watch.
Acquisitions
Apple is using its sizable bank account to acquire startups working in augmented reality. It’s through these acquisitions it will shape and perfect any future products. The most recent acquisition took place in August 2018 when Apple purchased Colorado-based Akonoia Holographics. The startup manufactures lenses for augmented reality glasses and holds more than 200 related patents according to Tom’s Guide.
On November 21, 2017, it was reported Apple acquired Vrvana, a startup that hit the headlines a few years back with its Totem headset. Apple paid around $30 million for the Montreal-based Vrvana, two sources claiming to have knowledge of the deal told TechCrunch. When the Totem landed on Kickstarter in 2014, it caught the attention of commentators for a number of reasons. The headset had onboard “pass-through” cameras to combine AR effects with the world around the wearer, head tracking, and the ability to show full-color animations, separating it from other headsets. The effect combined both VR- and AR-style experiences. Despite its promise, the Totem headset has never been released.
Apple hasn’t confirmed the acquisition, but nor has it denied it, and some former Vrvana employees are already working for the Cupertino-based company. The startup’s website is still online, though its social media updates ceased in the summer, suggesting a timeframe for when the reported acquisition may have taken place.
In 2016, Vrvana was known to be “planning a business-to-business model wherein it will sell the Totem hardware directly to enterprise customers in small volumes,” according to Tom’s Hardware. Whether Apple sees consumer potential for Vrvana’s technology or intends to also target businesses with any future augmented reality products and services remains to be seen.
Over the past three years, Apple has acquired companies with an expertise in AR, 3D mapping, and computer vision — including the 2013 purchase of PrimeSense, which pioneered the depth-tracking technology behind the Kinect, Metaio, and Flyby. In September 2016, Apple hired two VR veterans from Oculus VR and Magic Leap, companies with a pedigree in augmented reality and virtual reality technologies.
Will Apple’s AR headset work with the iPhone?
Will Apple’s AR headset need the iPhone to work, like Google’s Daydream View requires an Android phone? Apple may approach its AR device like the Apple Watch, with a dedicated processor and software, that can link to the iPhone, but doesn’t require it to operate.
In November 2016, Bloomberg reported that Apple was working on a pair of glasses that blended an augmented reality display with a standard pair of glasses. It was said to have begun talks with potential suppliers for components like near-eye displays, with the goal of developing a pair of glasses that could connect wirelessly to an iPhone and overlay information on the wearer’s field of vision.
Updated on August 30: Apple acquires Akonia Holographics.
