Ask me to fix a broken car or motorbike, and you would be out of luck.
But that is exactly what Microsoft wanted me to do. And the tool it gave me was its new mixed reality HoloLens 2 headset.
After I slipped it on, a set of digital instructions popped up over an actual broken ATV, set up in a mock repair shop at Microsoft’s headquarters. There were buckets of bolts to my right side and tools to my left. Then a set of neon-blue cartoonish arrows appeared, pointing at which gears I needed to move and where to move them to start the repair.
Next, I was told to get parts from bins around the room. The blue arrow directed me to the bin with the bolt I needed, then the washer to go with it, and then to a table with the right wrench.
Which size bolt? What type of wrench? I didn’t need to worry about that, because the HoloLens 2 guided me every step of the way. After about eight minutes, I went from cluelessly looking at a broken ATV to marveling at my first repair job. As Keanu Reeves says in the sci-fi epic The Matrix after getting fighting skills downloaded directly to his brain, ‘I know kung fu.’
This is the promise of Microsoft’s HoloLens: a computer that can help you do pretty much anything you’d use your hands for. It does this by sensing where it is in a building and then laying computer images on top of the real world you’re actually seeing. And unlike Microsoft’s original HoloLens, announced four years ago, this new headset is more comfortable and easier to use.
That makes working with HoloLens that much more immersive when you’re looking at the holograms it’s creating. And those can be anything from a massive movie screen to a model of a new building. Or a set of instructions showing you how to bake a batch of cookies or repair an aircraft engine. And with a pair of cameras, the headset can also be a phone-a-friend, allowing anyone to see through the eyes of whoever’s wearing the headset. Think about that. Your friends can also use an app to circle things they see, which show up as holographic notations when you’re wearing the headset.
‘The goal is these things will transform humans,’ HoloLens leader Alex Kipman said in an interview at Microsoft’s Redmond, Washington, headquarters. ‘They’ll empower people and organizations to do things they plainly were not able to do before.’
Microsoft isn’t crazy for thinking big about the potential of these headsets. The tech industry’s brightest minds are already investing billions of dollars to develop mixed reality (or, as some call it, augmented reality). They’re all betting it’ll change the way we use computers in the next few years. The market for AR gear is expected to explode, from under $6 billion last year to almost $200 billion by 2025, according to data compiled by Statista.
‘We don’t have to just imagine it, this future is here,’ Microsoft CEO Satya Nadella said at an event announcing the device Sunday at Mobile World Congress (MWC) in Barcelona. ‘Together, these advances are shaping the next phase of innovation.’
A high-profile startup called Magic Leap started shipping a $2,295 headset last August after working on it for seven years and raising $2.3 billion in investment from backers including AT&T, Google and Warner Bros. Facebook, which sells a $399 fully immersive virtual reality headset called the Oculus Rift, is working on a HoloLens competitor as well. Even Apple is secretly building one, which sources told CNET may arrive in 2020.
Microsoft is opening preorders for the HoloLens 2 on Feb. 24 for $3,500. The company plans to ship the device later this year.
HoloLens (the name is inspired by holograms) sounded straight out of sci-fi when it was first demoed. ‘We’re not talking about putting you into virtual worlds,’ Kipman said when he introduced the device in 2015. ‘We’re dreaming beyond virtual worlds, beyond screens, beyond pixels.’
But HoloLens had its problems. The original headset was hard to put on. It felt heavy at nearly 1.3 pounds, which is a little heavier than an iPad Air. The HoloLens squeezed some people’s heads, too. It didn’t always fit over your glasses. At $5,000, it was expensive. And the software used to run it often needed extra setup and adjustment.
The worst part was that the holograms appeared in an area in front of you that was only about the size of a deck of cards. (In AR parlance, that area’s called the field of view.) Practically, that meant you usually saw only a small portion of a hologram.
‘The previous headset’s field of view was its biggest drawback,’ said Scott Stein, CNET’s senior editor for reviews, who’s tried on more headsets than I knew existed. ‘Virtual objects have to be lined up just right, and the drop-off breaks the illusion and adds some fatigue trying to find things again.’
So Microsoft set out to fix those problems. Kipman invited Stein, CNET Español Managing Editor Gabriel Sama and me to the company’s headquarters to see what it’s accomplished with HoloLens 2. And to share the promise of HoloLens 3 and beyond.
Bottom line, according to Kipman: ‘We have a lot of work left ahead of us.’
Imagine walking through a store and instead of hunting for that bottle of mustard, arrows like the ones that helped me repair the ATV direct you through the maze of aisles, straight to the condiments on your grocery list.
Or they could display directions as you walk down the street.
Instead of arrows, a HoloLens 2 could show you what new furniture would look like in your living room. Or help you read a speech without a teleprompter.
For Kipman, this technology is a holy grail of computers. For the past three decades, we’ve been trapped looking at a screen while interacting with a keyboard, mouse or controller. Even smartphones are just tinier screens.
These types of computers are good enough for some things, like editing audio and video, coding an app or writing a story. But for nearly anything else, Microsoft thinks mixed reality is a better alternative.
That’s because researchers are finding that HoloLens may change the way we communicate, too. In one experiment at Stanford University’s Virtual Human Interaction Lab, a person was put in an empty room but through the HoloLens was able to talk to a photorealistic AR character. The avatar maintained eye contact, interacted naturally and gestured as if it were a real person.
‘It was jaw-dropping,’ said Jeremy Bailenson, a professor of communication who runs the lab.
Mixed reality systems like HoloLens and Magic Leap aren’t the only tech being lined up to revolutionize the way we use computers. There’s also virtual reality, in which headsets hold a screen so close to your eyes that you’re tricked into thinking you’re in the computer-generated world. After decades of VR development, Facebook CEO Mark Zuckerberg helped legitimize the technology when he spent more than $3 billion for then-startup Oculus.
Though the market is still nascent, people are using VR for immersive training (like showing a soldier what it’s like in a war zone), watching movies and playing video games. You can buy an Oculus Rift for $349, an HTC Vive for $499 or a Sony PlayStation VR for $299. They’re pretty much the same, though each offers screens of differing quality or slightly different controllers.
MR headsets, meanwhile, are radically different from company to company. The Magic Leap One system is made up of three devices, including a headset called Lightwear, which uses tech similar to that of HoloLens to overlay computer-generated images on the real world. But unlike HoloLens, Lightwear is powered by a hockey puck-size computer called the Lightpack, which has enough computing power to generate realistic-looking 3D images. The computer connects to the headset by a cord and has a slit in its middle, so it straddles your pocket and air can circulate around it and cool it down. You interact with the Magic Leap through a third gadget: a handheld remote called Control.
We know Apple is secretly developing a headset, but we don’t know much about its design, other than that it’s powered by Apple-designed chips, wirelessly talks to a computer and uses supersharp displays.
The only thing MR companies seem to agree on is that they aren’t selling to normal people — at least not yet. Magic Leap’s device was released to developers, with a consumer version promised for sometime in the not-too-distant future.
Microsoft says it’s focusing on business uses, even though it initially pitched HoloLens as a way to get work done and play games like its world-building phenom Minecraft. Now the company is saying HoloLens is an easy way to transport employees anywhere in the world. Or a way for a worker to do something complex without having to learn the process beforehand.
Kipman describes it as giving people superpowers. ‘This is a concept that’s been in our dreams,’ he says.
Finding a better fit
HoloLens 2 may not be ready for you and me, but it’s a markedly better device than its predecessor.
The first HoloLens was bulky. All the computing power was concentrated in front, along with the amalgam of cameras, motion sensors and microphones that help it understand where you are, what you’re doing with your hands and what you’re saying. It squeezed onto your head and, if you wore glasses like Stein or me, it was uncomfortable to wear for more than a few minutes.
To solve this problem, Microsoft turned to its Human Factors Engineering Lab. The group, housed in a nondescript building constructed on the Microsoft campus in 2015, and with $44 million in funding, was created to help the tech giant understand how to build devices to fit most people.
Carl Ledbetter, senior director of design for Microsoft’s Device Design Team, started my tour of the lab by having me pick up an oversize Xbox video game controller sitting on a table by the entrance. It was heavier than the Xbox controllers I’d used, and I found it harder to reach all the buttons.
‘You,’ he said, ‘are five years old.’
The oversize controller is a tool to help engineers understand all the people who use its devices, including kids. Microsoft took a similar approach with HoloLens 2, trying to outfit it for as many people as possible.
To improve on the original design, the team decided to evenly distribute HoloLens 2’s weight around your head by moving the battery and some other guts to the back in a pack about the size of a tin of Altoids. That helped make it comfortable — it was like slipping on a baseball cap.
Not all our heads are the same, though. So Microsoft spent a year scanning more than 600 people’s heads, using a machine with 36 interlinked cameras surrounding a seat to capture each form. Ultimately, these scans helped Microsoft understand different body types for men and women, short to tall, big to small. Some people have higher cheeks, lower noses or bigger bumps at the back of their head. Microsoft wanted the HoloLens 2 to fit 95 percent of them. (Sorry, Hodor.)
‘It’s not just a matter of scaling head sizes,’ Ledbetter said. ‘Everyone’s head is different.’
The HoloLens 2 also has sensors that scan your eyes. This helps it quickly identify who you are, so you can share a headset with multiple people but have it set up your way each time you put it on. The device also watches where you look, so an app can help you read by moving text along with you or merely respond to what you’re looking at.
And if you need to take the HoloLens off quickly to chat with someone in the real world, you flip it up, like putting a pair of glasses on your forehead.
‘Once the first mockup was built, it was like, ‘Wow, we’ve got to do that,” Ledbetter said. ‘It was a delighter.’
A new view
The most dramatic change with HoloLens 2, though, is the field of view, or how much hologram you see. In the first HoloLens, if you moved your head too much in any direction, the holograms would disappear.
For HoloLens 2, Microsoft more than doubled the area where you can see them. It did this by creating a new hologram technology.
In HoloLens 1, holograms were created by reflecting images from a tiny screen in the headset into specially made lenses for red, blue and green light waves. Those light waves were then beamed into the back of your eyes, where your brain would create the final image.
In HoloLens 2, the tiny screen has been replaced by a mirror known as a MEMS that moves fast enough to create the illusion of a screen in space. The MEMS creates 120 of these screens each second, filtered to your eye through new, sleeker lenses in the headset. The result is smooth movements, brighter animations and quick response if you move your head.
Most importantly, there’s much more area for the holograms to show up.
Before Microsoft started using MEMS technology, some executives believed it would be many years before the company could improve the HoloLens field of view. ‘It was a significant engineering problem,’ said Zulfi Alam, a general manager who worked on the displays and screens of the HoloLens.
Kipman called the new approach a ‘miracle.’
The result is that when I was fixing the ATV, I didn’t notice the holograms weren’t in all the areas I could see. There was just enough that I stopped thinking about it.
One reason Microsoft says HoloLens 2 isn’t good enough yet for everyday people is that there’s still not enough for us to do with it. Some people believe the killer app, or must-have feature, of HoloLens and Magic Leap will be when they replace all the screens in our homes. We’d just slap on a headset and see them as holograms.
Kipman isn’t convinced. It’s like thinking of a car as a ‘horseless buggy,’ he says. Instead, HoloLens is for those times a monitor and keyboard don’t work. That might be when you’re checking schematics of an oil pipeline in the field, or an X-ray of a patient overlaid on the body.
‘If you can take a worker that makes $150,000 and make him 40 percent more efficient, that’s $60,000 a year,’ said Jim Heppelmann, CEO of business software maker PTC, whose software works with the HoloLens to help workers in the field use schematics to repair broken machines, among other things. ‘I’ll buy $5,000 devices all day long.’
To encourage those business connections even further, Microsoft made several advancements to the software powering HoloLens. Among them, the company created software that uses the sensors on the headset to map buildings they’re in so that someone using an iPhone instead of a HoloLens can see the holograms in the room. HoloLens also works using technology from Microsoft’s Windows PC software, making it easier to code apps for and connect to the web.
Despite the excitement, this could still all go bust. Remember Google Glass?
When Glass was announced in 2012, people predicted that the smart eyewear would do things similar to what’s expected for HoloLens. Google mostly gave up on the project in 2015 after significant criticism over its usefulness, and privacy concerns raised by its attached camera.
Intel also had a smart-glasses initiative, which launched in 2013, but the company shut down the project last year after struggling to find partner companies. And Osterhout Design Group, whose smart glasses were the talk of CES in 2017, shut down, and its patents were sold last month.
These setbacks don’t worry Paul Bettner. He’s one of the people behind the hit mobile game Words With Friends, who later made an early high-profile VR game called Lucky’s Tale. Eventually, Bettner and other techies say, devices like the HoloLens will shrink down to the size of normal glasses, at which point they’ll become a part of everyday life.
But to get there, companies need to make incremental progress, just like they did with smartphones. ‘You couldn’t have gone from nothing to the iPhone,’ he said. ‘You needed to make flip phones first, then Palm Treos and then everything else to get there.’
HoloLens isn’t Kipman’s first attempt at changing the way we interact with computers. That was Project Natal, an accessory designed for Microsoft’s Xbox 360 video game console that could track your body as you moved in front of it. It did this by blanketing a room with thousands of infrared dots, and then using a camera to map where they land.
The Kinect, as it was later named, was so popular it sold 8 million units in its first 60 days, earning it a spot in the Guinness Book of World Records as the fastest-selling consumer gadget up to that point (Apple’s iPad nabbed the title in 2011).
‘The inspiration really was could we actually incorporate a human into the gameplay,’ Kipman said at the time. ‘Allowing you to just get in and play really inspires us.’
Microsoft ultimately sold 35 million Kinects, but the device fizzled out as developers struggled to find fun ways to use it in their games. Still, the tech helped inspire Face ID and animoji on Apple’s iPhones, face-unlock for Windows PCs and, of course, HoloLens.
Today, Microsoft’s enthusiasm for MR is contagious. Investment in software companies working on AR and MR for consumers is expected to hit $270.8 million this year, nearly double the investment in companies making similar VR software, according to Nielsen’s SuperData Research. Meanwhile, in the business world, companies like Chevron have bought hundreds of HoloLens headsets as they begin to use them widely among their employees.
As Microsoft gears up to start selling HoloLens 2, Kipman is looking ahead a couple of years to when he’ll show off HoloLens 3.
He demurred when asked about details, other than to promise it’ll be even more comfortable, easier to use and have more capabilities than what’s available now. Whatever the details are, Kipman said they’re what got the US Army interested enough to invest $480 million in more than 100,000 headsets to ‘increase lethality by enhancing the ability to detect, decide and engage before the enemy,’ according to a government filing. (The contract, however, has stirred dissent in the Redmond ranks, which Nadella has dismissed.)
HoloLens won’t be the only device vying for space on your head. There’s still Magic Leap, and Apple’s unannounced headset. By 2022, ‘tens of millions’ of these things will have been sold, said Tim Merel, managing director of market watcher Digi-Capital.
For now, though, Kipman plans to keep HoloLens just out of reach for you and me. ‘I have no interest in overhyping these products,’ he said. ‘There is a point where it will have enough immersion, enough comfort and enough out-of-box value where I will be happy to announce a consumer product.’
Until then, you’ll just have wait. Unless you need to repair an ATV.