In 1999, The Matrix introduced your neighbor, your dad, and pretty much every hacky-sacking college kid in the country to the idea that the real world around us … might not be so real.
In the film, our trenchcoated protagonist Neo discovers that the world as he knows it is only an illusion, piped into his brain while his body sits submerged in a gooey chemical broth. Trippy.
The idea that we are not really here at all — that life is just an illusion – is as old as Plato’s Allegory of the Cave. But The Matrix had that special sauce that made this mind-bending concept palatable to high schoolers shuffling around in JNCO jeans: guns, Keanu Reeves, and a soundtrack anchored by Rage Against the Machine. “Entering the Matrix” became pop-culture shorthand for the notion that technology could eventually deliver us from our mind-numbing reality and allow us to live in a faux universe of our own creation.
Want to learn kung fu in seconds? Freeze time? Drop bullets with a wave of your hand? Just take the red pill.
The Matrix has gone from a dystopian sci-fi dream to a waking reality.
A kid born in 1999 is just now old enough to rent the R-rated Matrix — or more likely, stream it. Yet in those intervening 17 years, entering the Matrix has gone from a dystopian sci-fi dream to a waking reality. These days, a pair of $800 goggles can convince you to duck as dinosaurs shamble over you, drop the pit of your stomach as you peer off the ledge of an artificial skyscraper, and make you puke — in real life — after one too many loops in a computer-generated space fighter.
And yup, you can freeze time and stop bullets, too.
As a society, we are glimpsing into the Matrix, and we like it. How did we get here, and what’s next? Here’s where VR came from, how it finally delivered after decades of failed promises, and how it’s going to utterly reshape our world in the next 10 years.
Virtual what now?
The 1992 film The Lawnmower Man introduced many Americans to the term “virtual reality” for the first timewith its tale of a scientist who uses the technology to accelerate the learning of a mentally handicapped gardener. But before the movie put the term in our vocabulary, Jaron Lanier was making it — well, reality.
Lanier was the inspiration behind Pierce Brosnan’s character in the movie, and popularized the term virtual reality back in 1989 when he developed the first VR headset for consumers, dubbed the EyePhone Model 1. He had high hopes.
“The thing that’s remarkably beautiful to me about virtual reality is that you can make up reality in virtual reality and share it with other people. It’s like having a collaborative lucid dream,” Lanier gushed in a 1989 interview for Whole Earth Review magazine. “It’s a world without limitations, a world as unlimited as dreams.” He envisioned humans leaving behind the shackles of the physical world to build, communicate, and experience new things in a parallel utopian universe where creativity was the only constraint. A $1,000 bill, he mused, was exactly the same as a $1 bill in virtual reality.
Except it still took a lot of those real-life $1,000 bills to make the whole thing run. The first iteration of VPL Research’s headset sold for $9,400, even without the ’roided-out supercomputers needed to power it. And a pair of Silicon Graphics supercomputers still struggled to make good on Lanier’s fantasies.
VR was a sham. Overhyped. It was an idea whose time had not yet come.
Upon visiting Lanier’s lab, Whole Earth Review editor Kevin Kelly struggled to dial in the vertical alignment of the goggles, then strained to see in 3D. “The goggles are also heavy,” he wrote, “leaving an indented line in your forehead when you take them off.”
Only a year later, Lanier dissolved the company and sold its patents, but the tech lived on. By the time Lawnmower Man hit theaters in 1992, you could experience VR right in the cinema lobby. Just hand over $5, don a neon headset, slump down into a plastic La-Z-Boy, and battle pterodactyls in stunning 3D inside a Virtuality pod.
Except the same barriers that plagued Lanier’s VR prototypes persisted in the shiny plastic commercial spinoffs. This was still the age of Wolfenstein graphics, not Call of Duty. And with a resolution of just 276 x 372 pixels in each eye, the screen of an old flip phone would look sharp in comparison.
When Nintendo’s catastrophic Virtual Boy finally blundered onto store shelves in 1995, to the universal disappointment of 10-year-olds everywhere, the fate of “virtual reality” was already cemented. VR was a sham. Overhyped. It was an idea whose time had not yet come.
Getting Luckey
More than a decade later, believers were still waiting.
“I’ve tried every damn video headset on the market — and I’m sure a few that never made it to market — and quite frankly, I thought they all sucked.” Ben Kuchera wrote in his 2007 review of the Vuzix iWear VR920. “It sounds like a sexy idea, but it’s rare that a headset is anything approaching tolerable.”
Then an enterprising 18-year-old from Long Beach, California, tried his hand. Palmer Luckey already had a passion for virtual reality, and a collection of lackluster headsets — including the Vuzix. So in 2010, he decided to hack together his own in the preferred laboratory of mad geniuses from California: his parents’ garage.
Luckey went from VR dabbler to VR wunderkind.
Twenty years of Moore’s Law had done wonders dissolving the obstacles of the ’90s. “Graphics, displays, the ability to compute complex geometries, the ability to track, a lot of those things came together because of the smartphone industry over the last five to 10 years.” Explains Andrew Nartker, Google VR product manager. “You had really high-density displays in really small, powerful computers, at very affordable price points, with tracking systems that basically were gyroscopes and accelerometers, packaged up in a form factor that everybody had access to.”
As Luckey cobbled together prototypes and chronicled every development on a forum for VR enthusiasts, he inadvertently attracted the attention of Doom creator and gaming legend John Carmack, who worked at Id Software. After swapping prototypes and code with the graphics luminary, Luckey went from VR dabbler to VR wunderkind. The world was clamoring for a headset by then named the Rift — so-called by Luckey because of the divide it created between the real and virtual worlds.
Things moved quickly. Just two months after creating the company Oculus to market his invention, Luckey launched a modest Kickstarter asking for $250,000 to fund production of a real prototype. Eager enthusiasts plunked down $2.4 million for access to the first batch of headsets, dubbed the Development Kit 1.
Carmack left Id Software to become Chief Technology Officer at Oculus in 2013. Facebook paid $2 billion for the company in 2014. Virtual reality was becoming real. But it was still missing something.
The Holodeck experience
While Oculus was minting celebrities and fawning press coverage, HTC was building the VR headset everyone would want.
The Taiwanese company best known for its smartphones may have seemed an unlikely contender in VR, but it had two advantages few competitors did: The ability to work incredibly fast, and a secret partnership with gaming giant Valve, which had its own jump-start on VR, but needed someone to build it.
“They had a prototype of room-scale tracked VR,” recalled Dan O’Brien, HTC’s vice president of VR. “The room is covered in fiducial markers; they look like big QR codes. They’re all over the ceiling, the walls, everything.” Valve’s prototype used the markers to determine its location in space, opening the entire room for the wearer of a headset to stride around and explore VR locations as if walking around the Holodeck from Star Trek. Using the Rift, you could merely stand still and look around.
It didn’t just work; it blew the world away.
HTC knew it was a game-changer, and struck a deal with Valve. “We established our partnership and said, ‘We’ve been building VR, but this is a different VR. This is actually like that promise of VR, putting somebody at the center of it and having it all around you, being able to move around really comfortably in the space,’” O’Brien said.
But the experience needed one more thing. “The first thing I did was stick my hands out,” O’Brien says of his initial demo. “There were no hands, there was no input, there were no controllers.” HTC had to find a way to control VR, and do away with the ugly QR codes that made room-scale work.
Shuttling engineers back and forth between Valve’s Bellevue, Washington, headquarters and HTC’s Taiwan headquarters, the team furiously iterated. They developed Wii Remote-like “sombrero” controllers to give players virtual hands, and replaced the QR codes with laser base stations that could invisibly track the headset and controllers.
“We were hot-gluing in the cables to try to keep them from falling out,” O’Brien admitted, laughing. By the time Mobile World Congress rolled around in March 2015, HTC’s team had a product it was proud to show off to the public for the first time. It didn’t just work; it blew the world away.
“HTC’s Vive is the most immersive, transformative virtual-reality experience I’ve ever had,” Digital Trends’ Jeffrey Van Camp raved upon trying the prototype at its unveiling in Barcelona, Spain. “Oculus Rift doesn’t hold a candle to this.” Despite a late start, HTC created a product better than Oculus’ Rift in every conceivable way.
What’s ahead?
You can buy an HTC Vive for $800 today, along with a $1,000 gaming PC, and unlock many of the experiences Jaron Lanier only dreamed of 25 years ago.
Space Pirate Trainer lets you gun down alien robots from the deck of a craft in the inky darkness of space. Google Earth lets you stoop down in your living room and inspect the striped rock layers of the Grand Canyon as if they were below your feet. Fantastic Contraption lets you build crazy wheeled machinations in your living room, as if you were toiling in some futuristic garage.
But the best experiences, VR acolytes agree, are still yet to come. Resh Sidhu leads VR development for Framestore, the high-end visual effects house that won an Oscar for the movie Gravity, and has since expanded into creating VR content. With hardware finally delivering on its promise, she believes it is now up to creatives to explore the possibilities.
(Credit: Valve)
“Anything I can imagine, I know I can reach out to a technology partner and they will find some way to make it,” she said. “If I want a haptic glove, if I want a whole body suit, if I need the Vive to have 60 people in a Vive experience, we can do that. From a creative point of view, it’s never been more exciting. The challenge now is to create content that is not about selling stuff in VR, but showing its true value.”
As it turns out, that means a lot more than just more games or movies. Pioneers in VR content see a range of world-changing uses that extend far beyond merely keeping us entertained.
Transplant your mind
What is it like to walk in someone else’s shoes? Books allow us to imagine it, and movies allow us to see it, but VR is the first medium that actually allows us to experience it. As VR developers catch on, generating empathy may turn out to be one of the medium’s most unique and powerful abilities.
Jeremy Bailenson, the founder of Stanford’s Virtual Human Interaction Lab, has spent years researching how VR may help us understand one another better, and the results are encouraging.
“After 20 minutes of being in it, I couldn’t talk to anyone for a little while. It was really powerful.”
“In a study where participants either became color-blind in VR, or simply imagined that they were color-blind, those who experienced the actual visual impairment in VR were likely to spend more time helping someone with the same disability after the study,” Bailenson told Digital Trends. “In another study, we found that having someone become a superhero made them more helpful in a real-life situation following the study.”
Howard Rheingold, who literally wrote the book on Virtual Reality in its early years, has visited the Stanford lab, and walked away a believer. “The research that Dr. Bailenson has done is peer reviewed, it’s real science,” Rheingold told us. “It’s astonishing. People report using less paper in their lives after they experience a VR experience in which, with the simulated use of paper, they can see forests being wiped out.”
These were academic studies, but Sidhu (from the Framestore special effects house) already sees the same emotional buttons being pushed outside the lab. Notes on Blindness is a documentary film with an accompanying VR experience that bowled her over. It tells the tale of a man losing his sight.
“After 20 minutes of being in it, I couldn’t talk to anyone for a little while. It was really powerful,” she said. “When you watch the film, it tells the same story, but nowhere near on the same emotional level. Nowhere near as much empathy. Nowhere near as much connection to that story and that individual. If that guy had been standing there, I would have reached out and given him a hug.”
Work with me here
Bailenson may be conducting research into VR with a focus on empathy, but he sees communications as the “killer app” for the medium. He’s not alone. Even back in the ’80s, Lanier dreamed of a world in which people used VR more like a telephone than a television.
“Observe somebody watching television. They look like a zombie. Then watch somebody using the telephone, and they look animated,” Lanier explained to Whole Earth Review. “The difference is that one is a social medium and the other is a broadcast medium.”
“If you were an architect and I’m a builder, we could be connecting and creating a building together, in real time.”
VR, he argued, was an inherently social medium as well.
We’re not there yet. Most of today’s VR experiences are isolated. But the future may bring virtual meeting places — Skype without the alienating effect of plastering your face on a washed-out screen in a conference room.
“Just think of what that means: Designers can connect all over the world,” Sidhu said. “If you were an architect and I’m a builder, we could be connecting and creating a building together, in real time, in the same space, and crafting and designing it.”
You don’t need to tell car companies twice. “They’re already developing their cars in VR, and making design decisions in VR,” HTC’s O’Brien explained. “Then they want to make collaborative decisions, because they have design teams all over the world.”
A group of designers from the United States, Spain, and Germany could pace around an upcoming design in real time discussing modifications, rather than pacing around a clay model doing the same thing while they fight jetlag after 10-hour flights.
As the world’s population booms, Rheingold sees environmental benefits to virtual meeting places, as well. “If one person out of 10 used VR instead of hopping in their car one day out of 10, that would have a significant impact,” he said.
Maybe the car companies won’t be as happy about that development.
Teach a man to fish
Remember what it was like when Apollo 11 landed humans on the moon? Unless you’re at least 50, you don’t. But Dan O’Brien’s 11-year-old son has a pretty good idea.
HTC’s VR guru recalls walking into his son’s room one day to find him with the VR headset on, sitting on the floor of his bedroom. Except to his son, it was a 1960s living room, and he was listening to JFK’s iconic “we choose to go to the Moon” speech on TV. A moment later, he was seated in the Apollo 11 module beside Buzz Aldrin. “I just let him go,” O’Brien said. “Afterwards, he came out, and he told me the whole thing that happened and what he learned. I didn’t have to ask him anything. This is going to change things.”
Atari creator Nolan Bushnell believes we’re essentially hardwired for this type of learning. “Our brains were created on the savannah when we were hunters and gathers,” he told Digital Trends. “All of our knowledge generation was situational. Whenever we have situational learning going on, our brains just take to it like peanut butter and chocolate. It’s the perfect mix.” Bushnell recently cofounded a company, Modal VR, that will provide “industrial-grade” VR to companies for exactly these types of experiences.
For students who struggle with learning from books, VR could be the ultimate “field trip,” opening up more engaging learning experiences that were too expensive, or even impossible, to access before. “It’s no longer about telling them, it’s about showing them,” Sidhu said.
“I don’t think we’re going to break education, but I sure hope that we’re going to evolve it in a very profound way.”
Teachers will need to incorporate lessons learned from game design, O’Brien argued, to make the most of the new medium. “You can’t just put someone in a virtual space, and then put them in a virtual classroom, and then just lecture to them there,” he said. “What you want to do is take what you learn from gameplay, and game designers, and actually introduce some of that gameplay into education.” Wandering around a Revolutionary War battlefield, in other words, may not impart the same lessons as fighting on it.
The opportunities don’t end with young people. VR could also help adults train for jobs, and help people in those jobs perform better. “Just as a microscope and a telescope extend our senses, so could VR.” Rheingold said. “Certainly for a chemist who’s trying to understand how protein molecules configure themselves in three-dimensional space, being able to actually manipulate active models of molecules would be useful. You might even come up with new compounds, new medications.”
O’Brien believes it could radically change the future. “I don’t think we’re going to break education,” he said, “but I sure hope that we’re going to evolve it in a very profound way. People are going to be able to experience a lot of things they’re learning about instead of just reading about it.”
Enter the Matrix
So when do we get the good stuff? When will virtual reality be indistinguishable from reality?
“Probably 30 years,” Bushnell suggested, unfazed. “We’re in the Pong stage of virtual reality. Basically, we hit photorealism in games almost 10 years ago. So in 35 years, we went from blocky quarter-inch pixels of Pong, to photorealism. It’s only reasonable to assume that we can do the same thing with virtual reality.”
That’s when real life goes full sci-fi.
“Once you get this stuff built, anybody can live the life of a rich person with just N number of calories a day to keep your body working,” Bushnell posited. “Then all that we have to do is worry about boredom. Do we now start fighting wars in virtuality? Is somebody going to sneak up on us when we’re all plugged in?”
The question of whether to take the red pill or blue pill, in other words, may be one you’ll have to answer in this lifetime. “Why would you not be in virtual reality?” Modal VR cofounder Jason Crawford asks. “You can have whatever reality you want.”
The societal implications could fill volumes. Who stays behind to run the simulation? How is it powered? Is a life lived in virtual reality as meaningful as one lived in reality? How do we get out? How do we know we’re not in it right now?
Bushnell and Crawford imagine some sort of test for that last one — like Cobb’s iconic spinning top in Inception. “Maybe that test is unhappiness,” Crawford suggests. Having a bad day means you’re in reality … because who would opt for a bad day in a universe of their own creation?
We might need that test sooner than you think — like now. “Elon Musk was chuckled at at [the Recode conference], and slightly ridiculed in some press, for saying there’s a one in billions chance that we’re not in a simulation,” Crawford says. “Nolan and I back Elon 100 percent. There’s a logical path to lead you to believe that argument.”
Reach out and touch someone
Before we get there, engineers have a lot of hard work to do.
Today’s headsets deliver a convincing visual display, but all it takes is lifting a weightless sword or walking through a wall to shatter the illusion. We still need the rest of our senses. And emulating the human sense of touch through haptic technology might be the hardest technology to master.
“In 10 years, [there will be] instances where visuals are passing convincibility and are on the path to indiscernibility.”
“I think we can simulate taste and smell pretty well,” Bushnell said. “But texture? In my mouth? The subtle changes as the enzymes break down the sugars into different things? Boy it’s going to be hard.”
Crawford believes we’ll need to communicate with nerve receptors directly to make it work, rather than fabricating mechanical imitations. “An inflatable bladder, and buzzes, and zaps, and pressure, and things like that … it’s too crude to trick our nerve endings.”
In other words, we need to talk to our nerves directly. But that means speaking their language, which scientists don’t entirely understand how to do yet. We know enough about the eyes to fake a vibrant yellow banana hanging from a tree in front of you, and enough about the nose to perfectly mimic its scent, but not yet enough to fool the hands into feeling its soft, rubbery texture. When we get there, not even the sky’s the limit. Nature goes out the window.
Just look at gum. “You can get a pack of gum at the store that is completely believable in some instances, and then you have these other flavors that are just crazy, flavors you never thought of!” Crawford said. “It’s going to be the same with all the other senses. Once you mimic them, then you can start to play with them and create things that people have never even thought of before.”
So future VR could not only allow us to feel the slick surface of ceramic or the soft fur of a cat, but totally synthetic touch sensations that don’t even exist in reality — like Juicy Fruit … for your hands.
The next 10 years of VR
Before we go full Matrix, most of us would probably settle for a headset that doesn’t have to be strapped to your face, and tethered by a thick bundle of wires. We’re headed in that direction fast.
“The headsets are going to be getting lighter,” HTC’s O’Brien assured us. “They’re going to be getting more comfortable. The audio is going to get better, the visual displays are going to get better. We will make those screens even brighter. We will make the resolution better.”
“Any technical power for the good can be perverted into something for the bad.”
Oh, and that unwieldy bundle of cords that spills out the back? It’s going away thanks to a clever technology you might not expect: eye tracking. By detecting what your eye is focused on, engineers could turn down the detail and refresh rates at the periphery of your vision, requiring less data to deliver the same visual quality. “We can actually start to improve throughput,” O’Brien explains. “So now we have less dependency on throughput and that cable.” Snip.
And the clunky black laser boxes that track you? “Let’s just make the tracking available in your house, instead of having a specific room and boxes,” O’Brien went on. “Let’s put the tracking in lightbulbs, or in the ceiling or something, and make it even easier.”
Advances in computing power will push graphics even further. “I believe that there will be, in 10 years, instances where visuals are passing convincibility and are on the path to indiscernibility,” ModalVR’s Crawford said.
And if you can’t personally afford VR yet, Bushnell would be happy for you to try one on somewhere else. “I believe that we’re going to see an explosion of mall VR arcades, or warehouses taken over for ‘the new paintball’ or ‘the new laser tag,’” Bushnell claimed. ModalVR will sell portable VR kits that can essentially transform any space into a VR space.
Google just wants to get everyone on board. “We’re really excited about simplifying the technology, simplifying the usability, making it understandable for anyone, so that they can understand why it would fit their life,” said Nartker, who helped launched the affordable Daydream View. He sees the Nintendo Wii as an inspiration. “I don’t think VR has had that type of moment yet, where it has gone from the enthusiast to the more mass-market audience.”
Content, Sidhu assured us, will also get better: “A lot of the live-action experiences that you see right now are passive: You’re like a ballerina in a music box. You can see the experience happening around you. In the future, we want to reach out and be part of that, and step out of the frame.” With the right cameras, rigs, and the processing power to stitch all the data together, she believes it’s already possible — we just need to connect the dots.
Of course, VR can get smarter, too. “I’ve recently been talking to IBM Watson about the way VR and AR will merge: A VR powered by artificial intelligence, experiences that are tailored to you, that know what you need, and are able to respond in real time,” Sidhu said. “That’s when it becomes exceptionally exciting. How this new medium can help us live our lives better. How it can help us become better humans.”
Will VR help shape a better society?
Can the ability to synthesize new realities, as Sidhu hopes, really help us become better humans, rather than just better-informed or better-entertained humans? Can it help shape a better society?
“Any technical power for the good can be perverted into something for the bad, whether we’re talking about nuclear power, whether we’re talking about telephones, whether we’re talking about social media,” Bushnell conceded. “I think that we have to be sensitive to the dark side, because there could be some really ugly things to it.”
But he also sees tremendous opportunities: Why not let criminals menace virtual worlds, rather than the real thing? And wouldn’t the unlimited abundance of a virtual world allow everyone to live as only the rich do now?
“I hope that there’s still human interaction, and we don’t go full Matrix.”
Of course, the overwhelming allure of virtual worlds could also be their fatal attraction. Do we really want everyone plugged in? “We already have a lot of people watching their TV sets, and playing video games, and looking at cute cat videos online,” Rheingold said. “I think that’s a legitimate fear, that people are going to be spending less time interacting with each other face to face, and more time interacting with each other virtually.”
For a man who confessed to us that a team member once slept while wearing the HTC Vive just to see how it felt to wake up in that realm, O’Brien admitted he doesn’t want this future, either. “I hope that there’s still human interaction, and we don’t go full Matrix,” he said. “That would be bad.” O’Brien hopes his son will grow up in a world where people still primarily interact face to face, even if they have the benefit of occasionally doing so virtually — or flying to the moon on an Apollo mission. “I constantly tell my son, “Hey! No more VR. Go out and play.’”
Even Lanier, the optimistic godfather of VR, had his skepticism. “I often worry about whether it’s a good technology or a bad technology,” Lanier confessed to Whole Earth Review all those years ago. “I’m sure bad things will happen with virtual reality; there will be some pain that it plays a part in because it will be a big thing and the world can be cruel. But I think overall it will actually have a tendency to enhance people’s sensitivity towards nature, towards preserving the Earth, because they’ll have a point of comparison.”
Taking vacations in virtual reality, in other words, might just be the breather that makes us better appreciate the splendor of the real thing.
